Communication terminal, display method, and non-transitory computer-readable medium

ABSTRACT

A communication terminal is communicable with another communication terminal mounted on a mobile apparatus or with a communication device of the mobile apparatus via a network. The communication terminal transmits operation instruction information for controlling the mobile apparatus to the another communication terminal or the communication device. The communication terminal includes circuitry configured to: receive a first video transmitted by one of the another communication terminal and the communication device and a second video having a wide-angle of view captured by a wide-angle image capturing apparatus; display, on a display device, at least one of the first video and the second video that are received; and display, on the display device, a controller for controlling an operation of the mobile apparatus as being superimposed on the at least one of the first video and the second video.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority under 35 U.S.C.§ 119(a) to Japanese Patent Application Nos. 2018-138860, filed on Jul.24, 2018 and 2019-014869, filed on Jan. 30, 2019, the entire disclosuresof which are incorporated herein by reference.

BACKGROUND Technical Field

Embodiments of the present disclosure relate to a communicationterminal, a display method, and a non-transitory computer-readablemedium.

Description of the Related Art

Communication systems are now in widespread use, according to whichcommunication terminals provided in different remote sites communicatevia a communication network such as the Internet or a local area network(LAN) to exchange video and audio data with each other. Suchcommunication systems allow users in the remote sites communicate. Forexample, users can hold a videoconference.

Further, robots to which the communication system as described above isapplied are known. Such robots are called “telepresence robots”. Inother words, the telepresence robot is a robot in which a“videoconference” and a “remote control technology” are combined. Thetelepresence robot uses the combination of remote control technologiesfrom a remote place and robot technologies, allowing an operator tooperate a mobile apparatus for moving a robot from a remote place andbehave as if the operator is present in a certain place.

Also, a technology for mounting a plurality of cameras on a robot isknown.

SUMMARY

Embodiments of the present disclosure describes a communicationterminal, communicable with another communication terminal mounted on amobile apparatus or with a communication device of the mobile apparatusvia a network. The communication terminal transmits operationinstruction information for controlling the mobile apparatus to theanother communication terminal or the communication device. Thecommunication terminal includes circuitry configured to: receive a firstvideo transmitted by one of the another communication terminal and thecommunication device and a second video having a wide-angle of viewcaptured by a wide-angle image capturing apparatus, the wide-angle imagecapturing apparatus being connected to one of the mobile apparatus andthe another communication terminal or being configured as a single unitwith one of the mobile apparatus and the another communication terminal;display, on a display device, at least one of the first video and thesecond video that are received; and display, on the display device, acontroller for controlling an operation of the mobile apparatus as beingsuperimposed on the at least one of the first video and the secondvideo.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

A more complete appreciation of the embodiments and many of theattendant advantages and features thereof can be readily obtained andunderstood from the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1A and FIG. 1B are schematic diagrams for describing an overview ofa communication system, according to an embodiment of the presentdisclosure;

FIG. 2A to FIG. 2D are illustration for describing display examples of afront-side video and/or a rear-side video captured by a planar imagecapturing apparatus and a spherical video captured by a wide-angle imagecapturing apparatus, according to an embodiment of the presentdisclosure;

FIG. 3 is a diagram illustrating an example of a configuration of thecommunication system, according to an embodiment of the presentdisclosure;

FIG. 4 is a block diagram illustrating an example of a hardwareconfiguration of a communication terminal, according to an embodiment ofthe present disclosure;

FIG. 5 is a block diagram illustrating an example of a hardwareconfiguration of a communication management system, according to anembodiment of the present disclosure;

FIG. 6 is a block diagram illustrating an example of a hardwareconfiguration of a mobile apparatus, according to an embodiment of thepresent disclosure;

FIG. 7 is a block diagram illustrating an example of a hardwareconfiguration of the wide-angle image capturing apparatus, according toan embodiment of the present disclosure;

FIG. 8 is an illustration of an example of how a user uses thewide-angle image capturing apparatus, according to an embodiment of thepresent disclosure;

FIG. 9A to FIG. 9C are illustrations for describing an overview of anoperation of generating an equirectangular projection image and aspherical image from images captured by the wide-angle image capturingapparatus, according to an embodiment of the present disclosure;

FIGS. 10A and 10B are illustrations for describing an overview of anoperation of generating an equirectangular projection image and aspherical image from images captured by the wide-angle image capturingapparatus, according to an embodiment of the present disclosure;

FIG. 11 is an illustration of relative positions of a virtual camera anda predetermined area in a case where the spherical image is representedas a surface area of a three-dimensional solid sphere, according to anembodiment of the present disclosure;

FIG. 12A is a perspective view of FIG. 11;

FIG. 12B is an illustration of a predetermined-area image displayed on adisplay, according to an embodiment of the present disclosure;

FIG. 13 is a view illustrating a relation between the predetermined-areainformation and the predetermined area, according to an embodiment ofthe present disclosure;

FIG. 14A and FIG. 14B are a block diagram illustrating an example of afunctional configuration of the communication management system and thecommunication terminals of the communication system, according to anembodiment of the present disclosure;

FIG. 15A and FIG. 15B are a sequence diagram illustrating an example ofprocesses in a communication preparation stage, performed by thecommunication system, according to an embodiment of the presentdisclosure;

FIG. 16 is an illustration of an example of a destination selectionscreen displayed on the communication terminal, according to anembodiment of the present disclosure;

FIG. 17A and FIG. 17B are a sequence diagram illustrating an example ofcommunication processes performed by the communication system, accordingto an embodiment of the present disclosure;

FIG. 18 is an illustration of an example of a video display screendisplayed on the display by the communication terminal in Pattern 1,according to an embodiment of the present disclosure;

FIG. 19 is a flowchart illustrating an example of an operation ofswitching a video to be displayed in a video layout example of Pattern1, according to an embodiment of the present disclosure;

FIG. 20 is an illustration of an example of the video display screendisplayed on the display by the communication terminal in Pattern 2,according to an embodiment of the present disclosure;

FIG. 21 is a flowchart illustrating an example of an operation ofdisplaying videos in a video layout example of Pattern 2, according toan embodiment of the present disclosure;

FIG. 22 is an illustration of an example of the video display screendisplayed on the display by the communication terminal in anotherexample display of Pattern 2, according to an embodiment of the presentdisclosure;

FIG. 23 is a flowchart illustrating an example of an operation ofdisplaying videos in the another display example of Pattern 2illustrated in FIG. 22, according to an embodiment of the presentdisclosure;

FIG. 24 is an illustration of an example of the video display screendisplayed on the display by the communication terminal in Pattern 3,according to an embodiment of the present disclosure;

FIG. 25 is a flowchart illustrating an example of an operation ofdisplaying videos in the video layout example of Pattern 3, according toan embodiment of the present disclosure;

FIG. 26 is an illustration of an example of the video display screendisplayed on the display by the communication terminal in anotherexample display of Pattern 3, according to an embodiment of the presentdisclosure;

FIG. 27 is a flowchart illustrating an example of an operation ofdisplaying videos in the another display example of Pattern 3illustrated in FIG. 26, according to an embodiment of the presentdisclosure; and

FIG. 28A and FIG. 28B are flowcharts for describing examples of triggersfor automatically switching the patterns, according to an embodiment ofthe present disclosure.

The accompanying drawings are intended to depict embodiments of thepresent disclosure and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise.

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

Hereinafter, a description is given of a communication terminal and amethod of displaying video performed by the communication terminalaccording to an example embodiment of the present disclosure, withreference to the drawings.

<Overview of Operation by Communication System and CommunicationTerminal>

Referring to FIG. 1A and FIG. 1B, a description is given of an overviewof a communication system 1. FIG. 1A is a schematic diagram illustratingan example of a configuration of the communication system 1 that uses atelepresence robot. The telepresence robot is provided in an office 801,and an operator 810 is present at home 802. The telepresence robotincludes a communication terminal 10B and a wide-angle image capturingapparatus 9, which are mounted on a mobile apparatus 20. The operator810 has a communication terminal 10A.

The wide-angle image capturing apparatus 9 transmits a spherical video(an example of a second video) described later to the communicationterminal 10B. Each of the communication terminal 10A and thecommunication terminal 10B include a planar image capturing device and amicrophone, and exchange video and audio with each other. Morespecifically, the communication terminal 10B transmits the sphericalvideo and a front-side video 831 (and/or a rear-side video 832, anexample of a first video) to the communication terminal 10A, and thecommunication terminal 10A transmits the front-side video 831 to thecommunication terminal 10B.

Further, in the communication terminal 10A, software (hereinafterreferred to as an “application”) that receives an operation relating tomovement or the like of the mobile apparatus 20 is operating. Theoperator 810 inputs an operation instruction while confirming the videotransmitted by the communication terminal 10B. The operation instructioninput by the operator 810 is transmitted from the communication terminal10A to the communication terminal 10B.

The communication terminal 10B and the mobile apparatus 20 cancommunicate by short-range communication such as Bluetooth (registeredtrademark), and the communication terminal 10B controls the mobileapparatus 20 based on the operation instruction transmitted from thecommunication terminal 10A. Accordingly, the operator 810 can cause themobile apparatus 20 to move from a remote place (home).

By causing the mobile apparatus 20 to move as described above, theoperator 810 can talk with a desired communication partner 820 asillustrated in FIG. 1B. FIG. 1B illustrates an example in which theoperator 810 causes the mobile apparatus 20 to move to a place where Mr.A 820 a is present and talks with Mr. A, and then causes the mobileapparatus 20 move to a place where Ms. B 820 b is present and talks withMs. B.

In addition to meeting persons, the operator 810 can perform any otheractivities without visiting an actual place, such as checking productsin remote stores, inspecting a factory, and participating inexhibitions, etc.

Since the planar image capturing apparatus that is built in thecommunication terminal 10B captures only a front-side image (video) anda rear-side image (video), it is difficult for the operator 810 torecognize the situation around the mobile apparatus 20 (e.g., an areaaround the bottom part of the mobile apparatus 20 or the area to left orright to the mobile apparatus 20). To address this issue, in the presentembodiment, the wide-angle image capturing apparatus 9 configured tocapture surroundings in 360-degree directions is mounted on thecommunication terminal 10B. In one example, the wide-angle imagecapturing apparatus 9 is mounted on the communication terminal 10B. Inanother example, the wide-angle image capturing apparatus 9 is mountedon the mobile apparatus 20. Since the wide-angle image capturingapparatus 9 can capture an image (video) of a wide-angle area, thewide-angle image capturing apparatus 9 is disposed on the upper sidewhere the wide-angle image capturing apparatus 9 is not likely to beblocked by surroundings. In the example of FIG. 1A and FIG. 1B, thewide-angle image capturing apparatus 9 is disposed on the upper side ofthe communication terminal 10B. In another example, the wide-angle imagecapturing apparatus 9 can be disposed on the left side or the right sideof the communication terminal 10B. In still another example, thewide-angle image capturing apparatus 9 can be disposed in the poleportion of the mobile apparatus 20.

Referring to FIG. 2A to 2D, a description is given of an overview of howthe communication terminal 10A according to the present embodimentdisplays a video captured by the planar image capturing apparatus and avideo captured by the wide-angle image capturing apparatus 9 configuredto capture omnidirectional 360-degree video. FIG. 2A to FIG. 2D areillustration for describing display examples of a front-side videoand/or a rear-side video captured by the planar image capturingapparatus and a spherical video captured by the wide-angle imagecapturing apparatus 9. In the present embodiment, a description is givenof three patterns as examples of layout of a front-side video, arear-side video, and a spherical video.

FIG. 2A illustrates a layout example of Pattern 1. Pattern 1 is a layoutexample in which a spherical video 840 and a front-side video (arear-side video) 841 are switched and displayed. In the left view ofFIG. 2A, the communication terminal 10A displays the spherical video 840in a full screen size. By contrast, in the right view of FIG. 2A, thefront-side video (or the rear-side video) 841 is displayed in a fullscreen size. When a certain image is displayed in a full screen size, itdoes not mean that any elements other than the certain image should notbe displayed. For example, an application software key or a screen ofanother application can be included. The operator 810 can switch betweenthe front-side video (or the rear-side video) and the spherical video.Accordingly, the operator can view both the front-side video (or therear-side video) and the spherical video on a large screen.

FIG. 2B illustrates a layout example of Pattern 2. Pattern 2 is a layoutexample in which the spherical video and the front-side video (rear-sidevideo) are displayed on the same screen. In FIG. 2B, an equirectangularprojection video 842, which is obtained by converting the sphericalvideo to planar video by equirectangular projection, is displayed in anupper smaller area, and the front-side video (or the rear-side video)843 is displayed in a lower larger area. This allows the operator 810 torecognize the front or the rear situation by viewing the front-sidevideo (or the rear-side video) 843 while recognizing the entiresituation by viewing the equirectangular projection video 842.

FIG. 2C and FIG. 2D illustrate layout examples of Pattern 3. Pattern 3is a layout example in which one of the spherical video and thefront-side video (rear-side video) is displayed as a main video, and amain screen and a plurality of sub screens are combined. In FIG. 2C, thespherical video is displayed on a main screen 844, and the front-sidevideo and the rear-side video are displayed respectively on a sub screen845 and a sub screen 846, which are arranged below the main screen. InFIG. 2D, the spherical video is displayed on a main screen 847, and aplurality of sub screens 848 are arranged over the main screen. On thesub screens 848, the front-side video, the rear-side video, andduplicates of a partial angle of view of the spherical video aredisplayed in a desired manner.

In FIG. 2A to FIG. 2D, a mobile apparatus operation button 614 (anexample of a controller) is displayed on at least one of the sphericalvideo and the front-side video (or the rear-side video). By using themobile apparatus operation button 614, the operator 810 can cause themobile apparatus 20 to move while viewing the spherical video or thefront-side video (or the rear-side video).

As described above, in the communication system according to the presentembodiment, the communication terminal 10A used by the operator 810appropriately arranges and displays the spherical video and thefront-side video (rear-side video). This allows the operator 810 tocheck an overview situation around the mobile apparatus 20 or check thefront-side video or the rear-side video having high resolution.

<Terms Used in This Disclosure>

The “operator 810” refers to a person who operates the mobile apparatus20. Although the operator 810 is also a user who uses the communicationterminal 10A, such user is referred to as an “operator” in the presentembodiment.

The communication partner 820 is a person in the office or the like. Inanother example, the communication partner can be an animal such as adog or a cat. In still another example, the mobile apparatus 20 can beused for any suitable purpose other than allowing the operator 810 totalk with the communication partner. For example, in a case where themobile apparatus 20 patrols an office or the like, the communicationterminal 10B just transmits video and audio to the communicationterminal 10A. In this case, the operator 810 has no conversation.

A place where the mobile apparatus 20 moves can be any place where themobile apparatus 20 can move. Further, the mobile apparatus 20 is notlimited to an apparatus that moves with power such as a motor. Forexample, a person can assist the movement of the mobile apparatus 20.

The “first video” refers to a video having a normal angle of view. The“second video” refers to a video having an angle of view that is widerthan the normal angle of view of the first video. The normal angle ofview is an angle of view in which distortion is acceptable even when aperspective projection lens is used for imaging. In other words, thefirst image can be said as a video imaged by a perspective projectionlens. The “video having a wide angle of view” refers to an imagecaptured by a wide-angle lens such as a fisheye lens, for example.Examples of projection method include stereographic projection,equidistant projection, isostatic projection and orthographicprojection. A spherical video obtained by capturing 360-degreesurroundings is one example of the video having a wide angel of view. Inanother example, the video having a wide-angle of view can be ahemispherical video or a video having an angle of view of 180 to 360degrees in the horizontal direction.

The term “large”, which is used in the description such as the firstvideo or the second video is displayed large, refers to that a displayarea of the first video or the second video is large, that is, the sizeof an area that occupies a screen. The number of pixels per unit lengthof the first video can be larger than that of the second video. Inanother example, the number of pixels per unit length of the secondvideo can be larger than that of the first video.

In the present embodiment, the term “video” is used on the assumptionthat it is a moving image. By contrast, the term “image” is used on theassumption that it is a single still image. However, since a videoincludes a plurality of images, and the term “video” and the term“image” are used in the present embodiment to describe any one of amoving image and a still image.

The description “perform(ing) control so that at least one of the firstvideo and the second video is displayed” refers to concurrentlydisplaying the first video and the second video, or displaying any oneof the first video and the second video. Preferably, this descriptionrefers to that a user can select whether the first video and the secondvideo are to be displayed concurrently or any one of the first video andthe second video is to be displayed.

The “controller for controlling a mobile apparatus” refers to, forexample, a user interface, a software key, a display component or thelike that can transmit some kinds of control signals to the mobileapparatus. A control signal for controlling the mobile apparatus to moveis one example of the control signals. In another example, the controlsignals include a control signal for controlling on and off of thepower, a control signal for controlling the stop and resume of imaging,a control signal for controlling illumination of an indicator lamp, anda control signal for controlling reproduction of a moving image.

<Example of System Configuration>

FIG. 3 is a diagram illustrating an example of a configuration of thecommunication system 1, according to an embodiment. The communicationsystem 1 includes a plurality of communication terminals (acommunication terminal 10A, a communication terminal 10B1, acommunication terminal 10B2, a communication terminal 10B3, acommunication terminal 10E), the mobile apparatus 20, a relay apparatus30, a communication management system 50. In the following description,any arbitrary one or more of the plurality of communication terminals(the communication terminal 10A, the communication terminal 10B1, thecommunication terminal 10B2, the communication terminal 10B3, thecommunication terminal 10E) is referred to as a “communication terminal10” or “communication terminals 10”. In addition, any arbitrary one ormore of the communication terminal 10B1, the communication terminal10B2, and the communication terminal 10B3 is referred to as a“communication terminal 10B” or “communication terminals 10B”. Thenumber of the communication terminals 10 illustrated in FIG. 3 is oneexample.

The communication terminals 10, the relay apparatus 30, and thecommunication management system 50 are communicably connected to oneanother through a communication network 2. The communication network 2includes, a local area network (LAN), the Internet, a mobile phonenetwork, and/or a dedicated line, for example.

Examples of the communication terminal 10 include a general-purposeinformation processing apparatus such as a tablet terminal, asmartphone, and a personal computer (PC) and a dedicated videoconferenceapparatus. Any one of the communication terminals 10 transmits andreceives image data, audio data or the like to and from one or more ofthe other communication terminals 10 to perform videoconference, forexample.

The communication terminal 10A executes an application supporting thecommunication system 1 to perform a videoconference with thecommunication terminal 10B and to remotely control the mobile apparatus20 via the communication terminal 10B. For example, the communicationterminal 10A can control the mobile apparatus 20 provided with thecommunication terminal 10B to move back and forth, left and right, etc.,by operating an operation button displayed on a display screen of thevideoconference. In one example, the communication terminal 10Aactivates browser software to cause the browser software to display avideo. In this case, the communication terminal 10A receive an operationinstruction for controlling the mobile apparatus 20, the operationinstruction being input to the browser software.

A combination of the mobile apparatus 20 and the communication terminal10B (including the wide-angle image capturing apparatus 9) is referredto as a “telepresence robot”. The mobile apparatus 20 is an apparatusthat drives a plurality of wheels in response to control from thecommunication terminal 10B attached to the mobile apparatus 20 toimplement a travelling function of performing movement such as “movingforward”, “moving backward”, “turning to the right”, and “turning to theleft”. The appearance of the mobile apparatus 20 illustrated in FIG. 3is just an example. The mobile apparatus 20 can be any suitableapparatus, provided that it can move along with the communicationterminal 10B in response to an operation instruction from thecommunication terminal 10B mounted on the mobile apparatus 20.

Further, the mobile apparatus 20 and the communication terminal 10B canbe constituted as a single unit. When it is said that the mobileapparatus 20 and the communication terminal 10B is constituted as asingle unit, what is meant is that the communication terminal 10B isattached to the mobile apparatus 20 before and after shipment, forexample. In addition to or in alternative to the above, when it that themobile apparatus 20 and the communication terminal 10B is constituted asa single unit, what is meant is that one can recognize from appearancethat the communication terminal 10B is a part of the mobile apparatus 20and that the color, design, manufacturer, and seller are common to thecommunication terminal 10B and the mobile apparatus 20, for example. Inaddition to or in alternative to the above, when it that the mobileapparatus 20 and the communication terminal 10B is constituted as asingle unit, what is meant is that the mobile apparatus 20 and thecommunication terminal 10B are inseparable or difficult to separate, oreven if they can be separated, some of functions are lost, for example.When the mobile apparatus 20 and the communication terminal 10B areconfigured as a single unit, the communication terminal 10B is treatedas a communication device of the mobile apparatus 20.

When the mobile apparatus 20 and the communication terminal 10B areconstituted as a single unit, the wide-angle image capturing apparatus 9can be also built in the mobile apparatus 20. Alternatively, even whenthe mobile apparatus 20 and the communication terminal 10B areconstituted as a single unit, the wide-angle image capturing apparatus 9as an external device can be attached to the mobile apparatus 20. Whenthe wide-angle image capturing apparatus 9 is built in the mobileapparatus 20, the wide-angle image capturing apparatus 9 is treated as awide-angle image capturing unit of the mobile apparatus 20.

In addition, even when the communication terminal 10B as an externaldevice is attached to the mobile apparatus 20, the wide-angle imagecapturing apparatus 9 can be built in the mobile apparatus 20.Alternatively, when the communication terminal 10B as an external deviceis attached to the mobile apparatus 20, the wide-angle image capturingapparatus 9 can be also attached as an external device to the mobileapparatus 20.

Further, the wide-angle image capturing apparatus 9 can be either builtin the communication terminal 10B or attached as an external device tothe communication terminal 10B. When the wide-angle image capturingapparatus 9 is built in the communication terminal 10B, the wide-angleimage capturing apparatus 9 is treated as a wide-angle image capturingunit of the communication terminal 10B.

The relay apparatus 30 is, for example, an information processingapparatus or a system including one or more information processingapparatuses. The relay apparatus 30 relays content data such as videodata, audio data, and operation instruction information exchanged amongthe plurality of communication terminals 10. Alternatively, theplurality of communication terminals 10 can directly exchange contentdata without the relay apparatus 30.

The communication management system 50 is, for example, an informationprocessing apparatus or a system including one or more informationprocessing apparatuses. For example, the communication management system50 performs login authentication in response to a login request from thecommunication terminal 10, management of the communication state of thecommunication terminal 10, management of a destination list, and controlof a session in which the plurality of communication terminals 10communicates with one another via the relay apparatus 30.

In one embodiment, a session is implemented by relaying content dataincluding image data and audio data (voice and other sounds) among theplurality of communication terminals 10 by the relay apparatus 30.

In the above configuration, the operator 810 communicates with thecommunication terminal 10B using the communication terminal 10A toremotely control the communication terminal 10B and the mobile apparatus20 to move. As a result, the operator 810 using the communicationterminal 10A can control the communication terminal 10B and the mobileapparatus 20 to move close to a desired communication partner andperform a videoconference or the like.

The communication system 1 includes, a data providing system in whichone communication terminal 10 transmits content data unidirectionallyfrom to the other communication terminal 10 via the communicationmanagement system 50, and a communication system in which a plurality ofcommunication terminals 10 exchange information, emotion, etc. with oneanother via the communication management system 50. This communicationsystem is a system that allows a plurality of communication terminals toexchange information, emotions, etc. with one another via the relayapparatus 30. Examples of the communication system include a televisionconferencing system, a videoconferencing system, and a videophonesystem.

<Hardware Configuration>

<<Hardware Configuration of Communication Terminal 10>>

FIG. 4 is a block diagram illustrating an example of a hardwareconfiguration of the communication terminal 10, according to anembodiment of the present disclosure. The communication terminal 10includes a configuration of a general-purpose computer. For example, thecommunication terminal 10 includes a central processing unit (CPU) 101,a read only memory (ROM) 102, a random access memory (RAM) 103, a flashmemory 104, and a solid state drive (SSD) 105. The communicationterminal 10 further includes a medium interface (I/F) 107, an inputdevice 108, a display 109, a network I/F 111, a camera 112, an imagingelement I/F 113, a microphone 114, a speaker 115, and an audioinput/output I/F 116. The communication terminal 10 further includes anexternal device connection I/F 117, a short-range wireless communicationdevice 118 and, a bus 119.

The CPU 101 is an arithmetic unit that reads out programs or data fromthe ROM 102, the flash memory 104 or the like, and executes processingaccording to the programs or data to implement functions of thecommunication terminal 10. The ROM 102 is a non-volatile memory in whichprograms such as an initial program loader (IPL) used for booting theCPU 101 are stored in advance. The RAM 103 is a volatile memory used asa work area for the CPU 101.

The flash memory 104 is a storage device that stores an operating system(OS), application programs, and various types of data. The SSD 105controls reading or writing of various data from or to the flash memory104 under control of the CPU 101. The medium I/F 107 controls, forexample, reading or writing (storing) data from or to a storage medium106, which is a recording medium such as a memory card. A program forcontrolling the communication terminal 10 can be stored in the recordingmedium.

The input device 108 is a device for receiving an input by an operatorof the communication terminal 10. Examples of the input device 108include a touch panel, a keyboard, and a pointing device. In anotherexample, the input device can be any suitable device that receives aninput by voice. The display 109 is a display device that display variousinformation to an operator of the communication terminal 10. The display109 and the input device 108 can be configured as a single unit, toimplement a display input device 110 such as a touch panel display.

The network I/F 111 is a communication interface that allows thecommunication terminal 10 to transmit and receive data through thecommunication network 2. The camera 112 includes an imaging element forcapturing an image of an object under control of the CPU 101. Theimaging element I/F 113 controls image capturing by the camera 112 andconverts data of the captured image into predetermined image data. Thecamera 112 is a planar image capturing device. The camera 112 capturesan image at an angle of view that is narrower than an angle of view atwhich the wide-angle image capturing apparatus 9 captures an image.However, the camera 112 captures an image with a resolution that ishigher than a resolution with which the wide-angle image capturingapparatus 9 captures an image. The term “resolution” refers to a degreerepresenting the fineness of a digital image. The resolution isexpressed by digitizing the fineness of individual unit points (dots,pixels) constituting the digital image. In general, “dot” is used as aunit representing resolution. The resolution of a display is oftenrepresented by the number of dots arranged horizontally and vertically,such as “1024×768 dots”. In a case where the wide-angle image capturingapparatus 9 and the camera 112 captures an image of the same imagingarea, the number of dots of the image captured by the wide-angle imagecapturing apparatus 9 is smaller than the number of dots of the imagecaptured by the camera 112.

The microphone 114 collects audio and converts the collected audio intoelectrical signals. The speaker 115 converts audio signals into audio,and outputs the audio. The audio input/output I/F 116 controls input andoutput of audio by the microphone 114 and the speaker 115. The externaldevice connection I/F 117 is an interface such as a universal serial bus(USB) for connecting the communication terminal to an external device.Examples of the external device connected via the external deviceconnection I/F 117 include the mobile apparatus 20 illustrated in FIG.3. The short-range wireless communication device 118 is a communicationinterface that controls communication of data with an external device(e.g., the mobile apparatus 20) by using a short-range wirelesscommunication network such as Bluetooth (registered trademark) orBluetooth Low Energy. The bus 119 is connected to each of theabove-described hardware elements, and transmits an address signal, adata signal, various control signals, or the like.

<<Hardware Configuration of Communication Management System 50 and RelayApparatus 30>>

FIG. 5 is a block diagram illustrating an example of a hardwareconfiguration of the communication management system 50, according to anembodiment. The communication management system 50 has a configurationof a general-purpose computer. For example, the communication managementsystem 50 includes a CPU 501, a ROM 502, a RAM 503, a hard disc (HD)504, a hard disc drive (HDD) 505, a media drive 507, and a display 508.The communication management system 50 further includes a network I/F509, a keyboard 511, a mouse 512, a compact disc-read only memory CD-ROMdrive 514, and a bus 510.

The CPU 501 is an arithmetic unit that reads out programs or data fromthe ROM 502, the HD 504 or the like, and executes processing accordingto the programs or data to implement functions of the communicationmanagement system 50. The ROM 502 is a non-volatile memory in whichprograms such as an IPL used for booting the CPU 501 are stored inadvance. The RAM 503 is a volatile memory used as a work area for theCPU 501.

The HD 504 is a storage device that stores an OS, programs such asapplication programs, and various types of data. The HDD 505 controlsreading or writing of various data from or to the HD 504 under controlof the CPU 501. The display 508 is a display device that displaysvarious information such as a cursor, menu, window, characters, orimage.

The network I/F 509 is a communication interface that allows thecommunication management system 50 to communicate data through thecommunication network 2. The keyboard 511 is one example of an inputdevice that receives inputs such as characters, numerical values,various instructions or the like according to an operation by a systemadministrator. The mouse 512 is one example of a pointing device thatreceives inputs such as selection and execution of various instructions,selection of a processing target, and movement of a cursor according toan operation by the system administrator.

The communication management system 50 and/or the relay apparatus 30does not necessarily always include the display 508, the keyboard 511and/or the mouse 512. In this case, the display 508, the keyboard 511and/or the mouse 512 can be connected to the communication managementsystem 50 and/or the relay apparatus 30 as needed.

The media drive 507 controls, for example, reading or writing (storing)data from or to a storage medium 506 such as a memory card. The CD-ROMdrive 514 controls reading or writing of data from or to a disc 513,which is one example of removable recording medium. The bus 510electrically connects the above-described hardware elements with oneanother such that an address signal, a data signal and various controlsignals are exchanged between the hardware elements.

The hardware configuration of the computer described above is just oneexample.

It is assumed that the relay apparatus 30 has the same hardwareconfiguration as that of the communication management system 50. Theprograms for the communication terminal 10, the relay apparatus 30 andthe communication management system 50 can be recorded in an installableor executable file format on a computer-readable storage medium fordistribution. Examples of the storage medium include a compact discrecordable (CD-R), a digital versatile disc (DVD), a Blu-ray disc, and aUSB memory. In addition, a storage medium such as a CD-ROM storing anyof the above-described programs and/or the HD 504 storing any of theabove-described programs can be distributed domestically or overseas asa program product.

<<Hardware Configuration of Mobile Apparatus 20>>

FIG. 6 is a block diagram illustrating an example of a hardwareconfiguration of a mobile apparatus 20, according to an embodiment. Themobile apparatus 20 includes, for example, a CPU 401, a RAM 402, a ROM403, an external device connection I/F 404, a short-range wirelesscommunication device 405, a wheel driving device 406, a steering device407.

The CPU 401 is an arithmetic unit that executes the program stored inthe ROM 403 or the like to implement functions of the mobile apparatus20. The RAM 402 is a volatile memory used as a work area for the CPU401. The ROM 403 is a nonvolatile memory storing data such as a programfor the mobile apparatus 20. Alternatively, the ROM 403 can be arewritable non-volatile memory such as a flash ROM.

The external device connection I/F 404 is a wired communicationinterface for connecting the mobile apparatus 20 to the external deviceconnection I/F 117 or the like of the communication terminal 10 by wire,to allow the mobile apparatus 20 to communicate with an external devicesuch as the communication terminal 10.

The short-range wireless communication device 405 is a wirelesscommunication interface that allows the mobile apparatus 20 to performwireless communication by the same wireless communication system as awireless communication system used by the short-range wirelesscommunication device 118 of the communication terminal 10, for example.The mobile apparatus 20 is configured to communicate with thecommunication terminal 10 via the external device connection I/F 404 orthe short-range wireless communication device 405, for example.

The wheel driving device 406 is one example of a driver that drives awheel or wheels to cause the mobile apparatus 20 to move. The wheeldriving device 406 includes, for example, a motor.

The steering device 407 is one example of a steering device that steersthe mobile apparatus 20 that is caused to move the wheel driving device406. For example, the steering device 407 changes the direction orinclination of the wheels. In another example, the steering device 407changes the direction of the mobile apparatus 20 by controlling thenumber of rotations or the speed of each of the left and right wheels.

<<Hardware Configuration of Wide-Angle Image Capturing Apparatus 9>>

Referring to FIG. 7, a hardware configuration of the wide-angle imagecapturing apparatus 9 is described. FIG. 7 is a block diagramillustrating an example of a hardware configuration of the wide-angleimage capturing apparatus 9, according to an embodiment. The followingdescribes a case in which the wide-angle image capturing apparatus 9 isa full-view spherical (omnidirectional) image capturing device havingtwo imaging elements. However, the wide-angle image capturing apparatus9 can include any suitable number of imaging elements, providing that itincludes at least two imaging elements. In addition, the wide-angleimage capturing apparatus 9 is not necessarily an image capturing devicededicated to omnidirectional image capturing. Alternatively, an externalomnidirectional image capturing unit can be attached to ageneral-purpose digital camera or a smartphone to implement an imagecapturing device having substantially the same function as that of thewide-angle image capturing apparatus 9.

As illustrated in FIG. 7, the wide-angle image capturing apparatus 9includes an imaging unit 301, an image processor 304, an imagingcontroller 305, a microphone 308, an audio processor 309, a CPU 311, aROM 312, a static random access memory (SRAM) 313, a dynamic randomaccess memory (DRAM) 314, an operation unit 315, a network I/F 316, acommunication device 317, an antenna 317 a, an electronic compass 318, agyroscopic sensor 319, and an acceleration sensor 320.

The imaging unit 301 includes two wide-angle lenses (so-called fisheyelenses) 302 a and 302 b, each having an angle of view of equal to orgreater than 180 degrees to form a hemispherical image. The imaging unit301 further includes two imaging elements 303 a and 303 b correspondingto the wide-angle lenses 302 a and 302 b respectively. Each of theimaging elements 303 a and 303 b includes an imaging sensor such as acomplementary metal oxide semiconductor (CMOS) sensor and acharge-coupled device (CCD) sensor, a timing generation circuit, and agroup of registers. The imaging sensor converts an optical image formedby the fisheye lenses 302 a and 302 b into electric signals to outputimage data. The timing generation circuit generates horizontal orvertical synchronization signals, pixel clocks and the like for theimaging sensor. Various commands, parameters and the like for operationsof the imaging elements 303 a and 303 b are set in the group ofregisters.

Each of the imaging elements 303 a and 303 b of the imaging unit 301 isconnected to the image processor 304 via a parallel I/F bus. Inaddition, each of the imaging elements 303 a and 303 b of the imagingunit 301 is connected to the imaging controller 305 via a serial I/F bussuch as an I2C bus. The image processor 304, the imaging controller 305,and the audio processor 309 are each connected to the CPU 311 via a bus310. Further, the ROM 312, the SRAM 313, the DRAM 314, the operationunit 315, the network I/F 316, the communication device 317, and theelectronic compass 318 are also connected to the bus 310.

The image processor 304 acquires image data from each of the imagingelements 303 a and 303 b via the parallel I/F bus and performspredetermined processing on each image data. Thereafter, the imageprocessor 304 combines these image data to generate data of anequirectangular projection image as illustrated in FIG. 9C.

The imaging controller 305 usually functions as a master device whileeach of the imaging elements 303 a and 303 b usually functions as aslave device. The imaging controller 305 sets commands or the like inthe group of registers of each of the imaging elements 303 a and 303 bvia the I2C bus. The imaging controller 305 receives various commandsfrom the CPU 311. Further, the imaging controller 305 obtains statusdata of the group of registers of each of the imaging elements 303 a and303 b via the I2C bus. The imaging controller 305 sends the obtainedstatus data to the CPU 311.

The imaging controller 305 instructs the imaging elements 303 a and 303b to output the image data at a time when the shutter button of theoperation unit 315 is pressed. In some cases, the wide-angle imagecapturing apparatus 9 is configured to display a preview image on adisplay (e.g., the display of the communication terminal 10) ordisplaying a moving image (movie). In case of displaying movie, imagedata are continuously output from the imaging elements 303 a and 303 bat a predetermined frame rate (frames per minute).

Furthermore, the imaging controller 305 operates in cooperation with theCPU 311, to synchronize the time when the imaging element 303 a outputsimage data and the time when the imaging element 303 b outputs the imagedata. In the present embodiment, the wide-angle image capturingapparatus 9 does not include a display. However, in another example, thewide-angle image capturing apparatus 9 can include a display (displayunit).

The microphone 308 converts sound into audio data (signals). The audioprocessor 309 obtains audio data output from the microphone 308 via anI/F bus and performs predetermined processing on the audio data.

The CPU 311 controls entire operation of the wide-angle image capturingapparatus 9 and performs necessary processing. The ROM 312 storesvarious programs for execution by the CPU 311. Each of the SRAM 313 andthe DRAM 314 operates as a work memory to store programs loaded from theROM 312 for execution by the CPU 311 or data in current processing. Morespecifically, in one example, the DRAM 314 stores image data currentlyprocessed by the image processor 304 and data of the equirectangularprojection image on which processing has been performed.

The operation unit 315 collectively refers to various operation keys,such as a shutter button. An operator operates the operation unit 315 toinput various image capturing (photographing) modes or image capturing(photographing) conditions.

The network I/F 316 collectively refers to an interface circuit such asa USB I/F that allows the wide-angle image capturing apparatus 9 tocommunicate with an external medium such as a secure digital (SD) cardor an external personal computer. The network I/F 316 supports at leastone of wired and wireless communications. The data of theequirectangular projection image, which is stored in the DRAM 314, isstored in the external medium via the network I/F 316 or transmitted toan external device such as the communication terminal 10B via thenetwork I/F 316, as needed.

The communication device 317 communicates data with an external devicesuch as the communication terminal 10B via the antenna 317 a of thewide-angle image capturing apparatus 9 through a short-range wirelesscommunication network such as Wi-Fi, near field communication (NFC), andBluetooth (registered trademark). The communication device 317 is alsoconfigured to transmit the data of equirectangular projection image toan external device such as the communication terminal 10B.

The electronic compass 318 calculates an orientation of the wide-angleimage capturing apparatus 9 from the Earth's magnetism to outputorientation information. This orientation information is an example ofrelated information, which is metadata described in compliance withExchangeable image file format (Exif). This information is used forimage processing such as image correction of captured images. Therelated information also includes data of a date and time when an imageis captured by the wide-angle image capturing apparatus 9, and a size ofimage data, for example.

The gyroscopic sensor 319 detects a change in tilt of the wide-angleimage capturing apparatus 9 (roll, pitch, yaw), for example, due tomovement of the wide-angle image capturing apparatus 9. The change intilt is one example of related information (metadata) described incompliance with Exif. This information is used for image processing suchas image correction of captured images.

The acceleration sensor 320 detects acceleration in three axialdirections. The attitude (an angle with respect to the direction ofgravity) of the wide-angle image capturing apparatus 9 is detected basedon the detected acceleration. Having the gyroscopic sensor 319 and theacceleration sensor 320, the wide-angle image capturing apparatus 9 cancorrect images with high accuracy.

<Spherical Video>

Next, referring to FIG. 8, a description is given of a situation wherethe wide-angle image capturing apparatus 9 is used. FIG. 8 illustratesan example of how a user uses the wide-angle image capturing apparatus9, according to one embodiment. As illustrated in FIG. 8, for example,the wide-angle image capturing apparatus 9 is used for capturing objectssurrounding a user who is holding the wide-angle image capturingapparatus in his or her hand. The imaging elements 303 a and 303 billustrated in FIG. 7 capture the objects surrounding the user to obtaintwo hemispherical images.

Next, referring to FIGS. 9A to 9C and FIGS. 10A and 10B, a descriptionis given of an overview of an operation of generating an equirectangularprojection image EC and a spherical image CE from images captured by thewide-angle image capturing apparatus 9. FIG. 9A is an illustration ofone of the two hemispherical images (front side), captured by thewide-angle image capturing apparatus 9. FIG. 9B is an illustration ofthe other one of the two hemispherical images (back side), captured bythe wide-angle image capturing apparatus 9. FIG. 9C is an illustrationof an image represented by equirectangular projection. The imagerepresented by equirectangular projection as illustrated in FIG. 9C is,hereinafter, referred to as an “equirectangular projection image” (or“equidistant cylindrical projection image”). FIG. 10A is a conceptualdiagram illustrating an example of how the equirectangular projectionimage maps to a surface of a sphere. FIG. 10B is an illustration of thespherical image.

As illustrated in FIG. 9A, an image captured by the imaging element 303a is a curved hemispherical image (front side) taken through the fisheyelens 302 a. Also, as illustrated in FIG. 9B, an image captured by theimaging element 303 b is a curved hemispherical image (back side) takenthrough the fisheye lens 302 b. The wide-angle image capturing apparatus9 combines the hemispherical image (front side) and the hemisphericalimage (back side), which are reversed by 180-degree from thehemispherical image (front side). This results in generation of theequirectangular projection image EC as illustrated in FIG. 9C.

The equirectangular projection image is mapped on the sphere surfaceusing Open Graphics Library for Embedded Systems (OpenGL ES) asillustrated in FIG. 10A. This results in generation of the sphericalimage CE as illustrated in FIG. 10B. In other words, the spherical imageCE is represented as the equirectangular projection image EC, whichcorresponds to a surface facing a center of the sphere CS. OpenGL ES isa graphic library used for visualizing two-dimensional (2D) andthree-dimensional (3D) data. The spherical image CE is either a stillimage or a moving image.

Since the spherical image CE is an image attached to the sphere surface,as illustrated in FIG. 10B, a part of the image may look distorted whenviewed from the user, providing a feeling of strangeness to the user. Toresolve this strange feeling, an image of a predetermined area, which isa part of the spherical image CE, is displayed as a planar image havingfewer curves. The predetermined area is, for example, a part of thespherical image CE that is viewable by the user. In this disclosure, theimage of the predetermined area is referred to as a “predetermined-areaimage”. Hereinafter, a description is given of displaying thepredetermined-area image, with reference to FIG. 11 and FIGS. 12A and12B.

FIG. 11 is an illustration of relative positions of a virtual camera ICand a predetermined area T in a case where the spherical image isrepresented as a surface area of a three-dimensional solid sphere CS.The virtual camera IC corresponds to a position of a point of view(viewpoint) of an operator who is viewing the spherical image CErepresented as a surface area of the three-dimensional solid sphere CS.FIG. 12A is a perspective view of FIG. 11. FIG. 12B is an illustrationof the predetermined-area image Q displayed on a display. In FIG. 12A,the spherical image CE illustrated in FIG. 11 is represented as asurface area of the three-dimensional solid sphere CS. Assuming that thespherical image CE is a surface area of the solid sphere CS, the virtualcamera IC is inside of the spherical image CE as illustrated in FIG. 12.The predetermined area T in the spherical image CE is an imaging area ofthe virtual camera IC. Specifically, the predetermined area T isspecified by predetermined-area information indicating an imagingdirection and an angle of view of the virtual camera IC in athree-dimensional virtual space containing the spherical image CE. Inaddition, zooming in the predetermined area T can also be determined bybringing the virtual camera IC closer to or away from the sphericalimage CE. The predetermined-area image Q is an image of thepredetermined area T, in the spherical image CE. The predetermined areaT is defined by the angle of view a and a distance f from the virtualcamera IC to the spherical image CE (see FIG. 13).

The predetermined-area image Q, which is an image of the predeterminedarea T illustrated in FIG. 12A, is displayed on a display as an image ofan imaging area of the virtual camera IC, as illustrated in FIG. 12B. Animage illustrated in FIG. 12B is the predetermined-area image Qrepresented by the predetermined-area information that is set bydefault. In the following description of the embodiment, an imagingdirection (ea, aa) and an angle of view a of the virtual camera IC areused. In another example, the predetermined-area image Q is identifiedby an imaging area (X, Y, Z) of the virtual camera IC, i.e., thepredetermined area T, rather than the angle of view α and the distancef.

Referring to FIG. 13, a relation between the predetermined-areainformation and the image of the predetermined area T is describedaccording to the embodiment. FIG. 13 is a view illustrating a relationbetween the predetermined-area information and the predetermined area T,according to one embodiment. As illustrated in FIG. 13, “ea” denotes anelevation angle, “aa” denotes an azimuth angle, and “α” denotes an angleof view. The position of the virtual camera IC is adjusted, such thatthe point of gaze of the virtual camera IC, indicated by the imagingdirection (ea, aa), matches a center point CP of the predetermined areaT as the imaging area of the virtual camera IC. As illustrated in FIG.13, when it is assumed that a diagonal angle of the predetermined area Tspecified by the angle of view α of the virtual camera IC is α, thecenter point CP provides the parameters (x, y) of the predetermined-areainformation. “f” denotes a distance from the virtual camera IC to thecenter point CP of the predetermined area T. “L” denotes a distancebetween the center point CP and a given vertex of the predetermined areaT (2L is a diagonal line). In FIG. 13, a trigonometric function equationgenerally expressed by the following equation 1 is satisfied.

L/f=tan(α/2)   (Equation 1)

<Functions>

Referring to FIG. 14A and FIG. 14B, the functions of the communicationsystem 1 are described. FIG. 14A and FIG. 14B are a block diagramillustrating an example of a functional configuration of thecommunication management system 50, the communication terminal 10B, andthe communication terminal 10A of the communication system 1.

<<Functional Configuration of Communication Terminal 10A>>

The communication terminal 10A receives an operation instruction forcontrolling a device such as the mobile apparatus 20. The communicationterminal 10A includes a data exchange unit 11, an operation inputreceiving unit 12, a communication control unit 13, an image capturingunit 14, an audio input unit 15 a, an audio output unit 15 b, a displaycontrol unit 16, and a data storage/read unit 17. These units arefunctions that are implemented by or that are caused to function byoperating any of the hardware components illustrated in FIG. 4 incooperation with the instructions of the CPU 101 according to theprogram for the communication terminal 10 expanded from the flash memory104 to the RAM 103. The communication terminal 10 further includes astorage unit 18, which is implemented by the RAM 103 illustrated in FIG.4 and the flash memory 104 illustrated in FIG. 4.

<<Functional Configuration of Communication Terminal 10B>>

The communication terminal 10B is a communication terminal 10 mounted onthe mobile apparatus 20. The communication terminal 10B is an example ofthe communication terminal 10 having a function of controlling a devicesuch as the mobile apparatus 20. The communication terminal 10Bincludes, in addition to the functional configuration of thecommunication terminal 10A described above, an operation instructionreceiving unit 19 a, a spherical video receiving unit 19 b, a devicecontrol unit 19 c, and an inter-device communication unit 19 d. Itshould be noted that the communication terminal 10A also includes theseadditional functions.

(Each functional Unit of Communication Terminal 10) A detaileddescription is now given of each of the functional units of thecommunication terminal 10 (communication terminal 10A and communicationterminal 10B).

The data exchange unit 11 transmits and receives various data (orinformation) to and from another (counterpart) communication terminal,device or system via the communication network 2. Before startingcommunication with a desired counterpart terminal, the data exchangeunit 11 starts receiving state information indicating the state of eachof the communication terminals as candidate counterparts, from thecommunication management system 50. The state information indicates theoperating state of each of the commination terminals 10, e.g., whetherthe communication terminal is online or offline. When the operatingstate is online, the state information further indicates a detailedstate such as whether the terminal is currently available forcommunication of is now currently communicating, or is currently busy(e.g., communicating with another counterpart).

The data exchange unit 11 of the communication terminal 10A receives thefirst video transmitted by the communication terminal 10B (or thecommunication device of the communication terminal 10B) and a sphericalvideo captured by the wide-angle image capturing apparatus 9, which isconnected to the communication terminal 10B or the mobile apparatus 20,or which is configured as a part of the communication terminal 10B orthe mobile apparatus 20.

The operation input receiving unit 12 receives various inputs to thecommunication terminal 10 from the operator. For example, when theoperator performs an operation of turning on the power of thecommunication terminal 10, the operation input receiving unit 12receives the operation and turns on the power.

In response to the reception of the power on operation, thecommunication control unit 13 controls the data exchange unit 11 toautomatically transmit, to the communication management system 50 viathe communication network 2, login request information indicating arequest for login and the current IP address of a request senderterminal, which is the communication terminal that sends the loginrequest information. In addition, when the operator performs anoperation of turning off the power of the communication terminal 10, thedata exchange unit 11 transmits, to the communication management system50, state information indicating that the power is to be turned off, andthen the operation input receiving unit 12 turns off the power.Accordingly, the communication management system 50 can detect that thepower of the communication terminal 10 is turned from on to off.

Further, the communication control unit 13 performs variouscommunication controls such as establishment and disconnection of asession which the communication terminal 10 exchanges content data withone or more of the other communication terminals via the relay apparatus30. The communication control unit 13 according to the presentembodiment includes a communication identification (ID) of thecommunication terminal 10 in session control information (e.g., startrequest information, start response information, etc., described below),and transmits the session control information including thecommunication ID to the communication management system 50.

The communication ID is an example of identification information of anaccount that can participate in a session in which the communicationterminals 10 exchange content data with one another. Examples of thecommunication ID include a user ID as identification information of theoperator, an application ID as identification information of anapplication, a contract ID as identification information of a contractorof the communication terminal 10. For example, a combination of at leasttwo of character(s), number(s), symbol(s), and various marks can be usedas the communication ID. In another example, an email address can beused as the communication ID.

The image capturing unit 14 converts image data obtained by imaging anobject into predetermined image (video) data and outputs the image(video) data. Two imaging capturing units 14 are provided, one beingprovided on the front side of the communication terminal 10 and theother one being provided on the back side of the communication terminal10. The image capturing unit 14 provided on the front side is used forcapturing a front-side video. The image capturing unit 14 provided onthe back side is used for capturing a rear-side video. There may bethree or more image capturing units 14.

After voice sound of the operator is converted to audio signals by themicrophone 114, the audio input unit 15 a converts the audio signals topredetermined audio data and outputs the audio data. The audio outputunit 15 b converts audio data into audio signals and outputs the audiosignals to the speaker, to control the speaker 115 to audiocorresponding to the audio signals.

The display control unit 16, for example, controls the display 109 orthe display input device 110 to display an image (video) based on imagedata included in content data received by the communication terminal 10.Further, the display control unit 16 transmits information of adestination list received from the communication management system 50 tothe display 109, to control the display 109 to display the destinationlist.

The display control unit 16 of the communication terminal 10A performscontrols so that at least one of a front-side video and a rear-sidevideo received by the data exchange unit 11 is displayed, and displaysthe mobile apparatus operation button 614 for controlling the mobileapparatus 20 as being superimposed on the front-side video or therear-side video.

The data storage/read unit 17 stores various types of data in thestorage unit 18 or reads out various types of data stored in the storageunit 18.

The storage unit 18 stores authentication information such as theabove-described communication ID and a password corresponding to thecommunication ID. Further, image data and audio data currently stored inthe storage unit 18 are overwritten with new image data and audio dataevery time the new image data and audio data are received in performingcommunication with a counterpart terminal. The display 109 displays animage based on the image data currently stored, and the speaker 115outputs audio based on the audio data currently stored.

A description is now given of each of the functional unit of thecommunication terminal 10B.

The spherical video receiving unit 19 b receives an equirectangularprojection video from the wide-angle image capturing apparatus 9 bywireless communication such as Bluetooth (registered trademark) or wiredcommunication such as a USB cable. The equirectangular projection videois a moving image that is repeatedly transmitted with a frequency thatcan be regarded as a moving image. In another example, theequirectangular projection image can be a still image. In still anotherexample, a moving image and a still image can be switched.

The operation instruction receiving unit 19 a receives, via the dataexchange unit 11, operation instruction information for requestingcontrol a device (mobile apparatus 20) to the communication terminal 10Bfrom the communication terminal 10A. The operation instructioninformation includes, for example, a communication ID of thecommunication terminal 10 that transmits the operation instructioninformation, and an operation instruction indicating the control contentto be requested.

The device control unit 19 c controls the mobile apparatus 20 based onthe operation instruction included in the operation instructioninformation received by the operation instruction receiving unit 19 a.The inter-device communication unit 19 d communicates with the mobileapparatus 20 by using the short-range wireless communication device 118.

<Functional Configuration of Mobile Apparatus 20>

The mobile apparatus 20 includes, for example, an inter-devicecommunication unit 21 and a traveling control unit 22. The inter-devicecommunication unit 21 is implemented by, for example, the externaldevice connection I/F 404 or the short-range wireless communicationdevice 405 illustrated in FIG. 6, each of which operates according toinstructions of the CPU 401. In the present embodiment, it is assumedthat the inter-device communication unit 21 communicates with thecommunication terminal 10B by using the short-range wirelesscommunication device 405.

The traveling control unit 22 controls the wheel driving device 406 andthe steering device 407 illustrated in FIG. 6 based on the controlcontent acquired from the communication terminal 10A, to controlmovement (travelling) of the mobile apparatus 20 such as moving forward,moving backward, turning to the left, and turning to the right.

<<Functional Configuration of Wide-Angle Image Capturing Apparatus 9>>

The wide-angle image capturing apparatus 9 includes, for example, awide-angle image transmitting unit 31 and a wide-angle image capturingunit 32. The wide-angle image capturing unit 32 captures a 360-degreewide-angle equirectangular projection video at a predetermined framerate. In another example, the wide-angle image capturing unit 32 cancapture a still image. The wide-angle image transmitting unit 31, whichis implemented by the communication device 317 illustrated in FIG. 7,transmits an equirectangular projection video as a moving image or astill image to the communication terminal 10B.

<<Functional Configuration of Communication Management System 50>>

The communication management system 50 includes a data exchange unit 51,a terminal authentication unit 52, a terminal management unit 53, aterminal extraction unit 54, a session management unit 55, a datastorage/read unit 57. These units are functions that are implemented byor that are caused to function by operating any of the hardwarecomponents illustrated in FIG. 5 in cooperation with the instructions ofthe CPU 501 according to the program for the communication managementsystem 50 expanded from the HD 504 to the RAM 503. The communicationmanagement system 50 further includes a storage unit 5000 implementedby, for example, the HD 504 illustrated in FIG. 5.

(Each Functional Unit of Communication Management System 50)

A detailed description is now given of each of the functional units ofthe communication management system 50. The data exchange unit 51transmits and receives various data (or information) to and from another(counterpart) communication terminal, device or system via thecommunication network 2.

The terminal authentication unit 52 performs authentication of thecommunication terminal 10 by determining whether a set of thecommunication ID and the password included in the login requestinformation received via the data exchange unit 51 is stored in anauthentication management database (DB) 5002.

The terminal management unit 53 stores, in a terminal management DB5003, for each of the communication IDs, a destination name, anoperating state, a date and time when the request information or like isreceived, and an IP address of a request sender terminal, which is acommunication terminal that sends the request information, inassociation with one another. For example, the terminal management unit53 changes the operating state of the terminal management DB 5003 fromOnline to Offline, based on state information indicating that the poweris turned off, the state information being transmitted from thecommunication terminal 10 in response to the operator's operation ofturning the power of the communication terminal 10 from on to off.

The terminal extraction unit 54 searches a destination list managementDB 5004 using the communication ID of the a request sender terminal thathas sent a login request as a search key, to extract the communicationID of a counterpart terminal that can communication with the requestsender terminal. Further, the terminal extraction unit 54 searches thedestination list management DB 5004 to extract the communication ID ofanother (counterpart) communication terminal that has registered thecommunication ID of the request sender terminal as a candidate of thecounterpart terminal.

Furthermore, the terminal extraction unit 54 searches the terminalmanagement DB 5003 using the extracted communication ID of thecounterpart terminal candidate as a search key, to read out theoperating state associated with each extracted communication ID.Accordingly, the terminal management unit 53 can obtain the operatingstate of the counterpart terminal candidate that can communicate withthe request sender terminal that has sent a login request. Further, theterminal management unit 53 searches the terminal management DB 5003 byusing the communication ID of the request sender terminal as a searchkey, to acquire the operating state of the request sender terminal thathas sent a login request.

The session management unit 55 controls a session managed by thecommunication management system 50. The control of session includes, forexample, control for establishing a session, control for causing thecommunication terminal 10 to participate in the established session,control for disconnecting the session, and generation of a session ID.In addition, the session management unit 55 stores, in a sessionmanagement DB 5005, the communication ID of the communication terminal10 that has requested the start of a session, the communication ID ofthe counterpart terminal, in association with a session ID, which isidentification information of a session.

The data storage/read unit 57 is implemented by the HDD 505 illustratedin FIG. 5 according to instructions of the CPU 501. Alternatively, thedata storage/read unit 57 is implemented by instructions of the CPU 501.The data storage/read unit 57 stores various types of data in thestorage unit 5000 and reads out various types of data from the storageunit 5000.

<Example of Information Managed by Communication Management System 50>

A description is now given of each of the management DBs stored in thestorage unit 5000 of the communication management system 50.

TABLE 1 Authentication Management Table: Communication ID Password 01aaaaaa 01ab abab 01ba baba

Table 1 is an example of an authentication management table 602 includedin the authentication management DB 5002 stored in the storage unit 5000of the communication management system 50. The authentication managementtable 602 stores communication IDs of the communication terminals 10managed by the communication management system 50 in association withcorresponding passwords. For example, the authentication managementtable 602 of Table 1 indicates that the password corresponding to thecommunication terminal 10 identified by the communication ID “01aa” is“aaaa”.

TABLE 2 Terminal Management Table: Communi- Reception cation DestinationOpaerating Date and IP Address ID Name State Time of Terminal 01aaJapan, Head Online 20xx.4.10 1.2.1.3 office, (Available for 13:40 AAterminal communication) 01b1 United States, Online 20xx.4.10 1.2.1.4First research (Available for 13:40 laboratory, communication) ABterminal 01b2 United States, Online 20xx.4.10 1.2.1.5 NY company-(Available for 9:50 owned store, communication) AC terminal 01b3 China,First Online 20xx.4.10 1.2.1.5 factory, (Communicating) 11:42 ADterminal . . . . . . . . . . . . . . .

Table 2 is an example of a terminal management table 603 included in theterminal management DB 5003 stored in the storage unit 5000 of thecommunication management system 50. The terminal management table 603stores, for each of the communication IDs of the communication terminals10, the communication ID of the communication terminal, a destinationname in a case where the communication terminal 10 serves as acounterpart terminal, the operating state of the communication terminal10, a date and time when login request information described below isreceived at the communication management system 50, and the IP addressof the communication terminal 10, in association with one another. Forexample, the terminal management table 603 of Table 2 indicates that aterminal name of the communication terminal 10 identified by thecommunication ID “01aa” is “Japan headquarters” and the operating stateof this communication terminal is “Online (available for communication).Further, this terminal management table 603 indicates that the date andtime when the communication management system 50 receives the loginrequest information from the communication terminal 10 identified by thecommunication ID “01aa” is “13:40 on April 10, 20xx” and the IP addressof this communication terminal is “1.2.1.3”.

TABLE 3 Destination List Management Table: Communciation ID ofCommunication ID of Counterpart Request Sender Terminal TerminalCandidate 01aa 01b1, 01b2, 01b3 01b1 01aa 01b2 01aa 01b3 01aa . . . . .. 01bd 01ab, 01ba, . . . , 01da, 01ca, 01cb, . . . , 01da

Table 3 is an example of a destination list management table 701included in the destination list management DB 5004 stored in thestorage unit 5000 of the communication management system 50. Thedestination list management table 701 stores the communication IDs ofrequest sender terminals, each of which sends a request for startingcommunication in videoconference in association with the communicationIDs of all counterpart terminals registered as counterpart terminalcandidates. For example, the destination list management table 701 ofTable 3 indicates that counterpart terminal candidates to which arequest sender terminal identified by the communication ID “01aa” cansend a request for starting communication are the communicationterminals identified by the communication IDs “01b1”, “01b2” and “01b3”,respectively. The counterpart terminal candidates stored in thedestination list management table 701 are updated by addition ordeletion performed by the communication management system 50 in responseto an addition or deletion request transmitted from an arbitrary requestsender terminal to the communication management system 50.

With this configuration, the request sender terminal (for example, thecommunication terminal identified by “01aa”) can start communicationonly with the counterpart terminal candidate (for example, thecommunication terminal identified by “01b1”) registered in advance. Thiscounterpart terminal (for example, the communication terminal identifiedby “01b1”) is not allowed to communicate with the request senderterminal (for example, the communication terminal identified by “01aa”)unless this request sender terminal is registered in the destinationlist management table 701 as a counterpart terminal. This configurationis preferable because the possibility of communication betweenunexpected communication terminals 10 can be reduced. Alternatively, anarbitrary communication terminal 10 can communicate with a desiredanother (counterpart) communication terminal without in-advanceregistration in the destination list management table 701.

TABLE 4 Session Management Table: Communication Session RelayCommunication ID of Particpation Session Apparatus ID of RequestCounterpart Date and ID ID Sender Terminal Terminal Time se1 111a 01aa01b1 20xx/04/10 13:45:30 01b2 20xx/04/10 13:50:30 se2 111a 01ad 01ca20xx/04/10 13:11:11 . . . . . . . . . . . . . . .

Table 4 is an example of a session management table 702 included in thesession management DB 5005 stored in the storage unit 5000 of thecommunication management system 50. The session management table 702stores, for each session ID as identification information of a session,a relay apparatus ID of the relay apparatus 30 to be used for relayingdata, the communication ID of a request sender terminal, thecommunication ID of a counterpart terminal, and a session participationdate and time, in association with one another. For example, the sessionmanagement table 702 of Table 4 indicates that a session identified bythe session ID “se2” is performed between a request sender terminalidentified by the communication ID “01ad” and a counterpart terminalidentified by the communication ID “01ca”. This session management table702 further indicates that the session identified by the session ID“se2” is started at “20xx/4/10, 13:11:11” via the relay apparatus 30identified by a relay apparatus ID “111b”.

<Processes>

A description is now given of processes in an operation performed by thecommunication system 1.

(Preparation Processes)

FIG. 15A and FIG. 15B are a sequence diagram illustrating an example ofprocesses in a communication preparation stage, performed by thecommunication system 1. In the present embodiment, a description isgiven of processes in a preparation stage performed before a sessionbetween the communication terminal 10A and the communication terminal10B1 is started, as an example. The following description is provided onthe assumption that the communication ID of the communication terminal10A is “01aa” and the communication ID of the communication terminal10B1 is “01b”.

First, when an operator of the communication terminal 10A, which is arequest sender terminal, performs an operation of turning on the powerof the communication terminal 10A, for example, the operation inputreceiving unit 12 receives this operation of turning on the power andturns of the power of the communication terminal 10A (step S21).

In response to the power being turned on as above, the communicationcontrol unit 13 transmits login request information for requesting alogin from the data exchange unit 11 to the communication managementsystem 50 via the communication network 2 (step S22). The operation ofturning on the power of the communication terminal 10A is just oneexample of a trigger by which the communication control unit 13transmits the login request information. In another example, the loginrequest information can be transmitted in response to an operation tothe input device 108 by the operator. In still another example, thelogin request information can be transmitted in response to activationof an application.

The login request information includes a communication ID foridentifying the communication terminal 10 a, which is a local terminalas the request sender terminal, and a password. These communication IDand password are information read from the storage unit 18 by the datastorage/read unit 17, for example. In addition, when the login requestinformation is transmitted from the communication terminal 10A to thecommunication management system 50, the communication management system50, which is a receiving side, receives an IP address of thecommunication terminal 10 a, which is a transmitting side.

Next, the terminal authentication unit 52 of the communicationmanagement system 50 searches the above-described authenticationmanagement table 602 by using the communication ID and the passwordincluded in the login request information received via the data exchangeunit 51 as search keys. The terminal authentication unit 52 performsauthentication based on whether a set of the communication ID and thepassword included in the login request information received from thecommunication terminal 10A is stored in the authentication managementtable 602 (step S23).

When the terminal authentication unit 52 determines that a requestindicated by the login request information is a login request from thecommunication terminal 10A having a valid usage authorization, theterminal management unit 53 changes the operating state that is storedthe terminal management table 603 in association with the communicationID “01aa” of the communication terminal 10A “Online (available forcommunication)”. At this time, the terminal management unit 53 updatesthe reception date and time, and updates the IP address of thecommunication terminal 10 as needed (step S24). As a result, in theterminal management table 603, the operating state “Online (availablefor communication)”, the reception date and time “20xx.4.10, 13: 40”,and the IP address “1.2.1.3” of the communication terminal 10A arestored in association with the communication ID “01aa” of thecommunication terminal 10A.

The data exchange unit 51 of the communication management system 50transmits authentication result information indicating an authenticationresult obtained by the terminal authentication unit 52 to thecommunication terminal 10A, which is a request sender terminal that hassent the above-mentioned login request information, via thecommunication network 2 (S25). In the following, the description isgiven of a case where in step S23, the terminal authentication unit 52determines that the communication terminal 10A is a communicationterminal that has a valid usage authorization.

The terminal extraction unit 54 of the communication management system50 searches the destination list management table 701 by using thecommunication ID “01aa” of the request sender terminal (thecommunication terminal 10A) from which the login request is received asa search key. By this searching, the terminal management unit 53extracts communication IDs of counterpart terminal candidates that cancommunicate with the request sender terminal (communication terminal10A) (step S26). For example, “01b1”, “01b2”, and “01b3” are extractedas the communication IDs of the counterpart terminals associated withthe communication ID “01aa” of the request sender terminal (thecommunication terminal 10A).

Next, the terminal extraction unit 54 searches the terminal managementtable 603 by using the extracted communication IDs (“01b1”, “01b2”,“01b3”) of the counterpart terminal candidates as search keys. By thissearching, the terminal extraction unit 54 reads out the operating stateassociated with each of the communication IDs (“01b1”, “01b2”, “01b3”)to acquire the operation state of each of the communication terminalsidentified by these communication IDs (step S27).

Next, the data exchange unit 51 transmits, to the request senderterminal (communication terminal 10A), counterpart state informationincluding the communication IDs (“01b1”, “01b2”, “01b3”) of thecounterpart terminal candidates and the operating state of each of thecommunication terminals identified by these communication IDs (stepS28). This allows the request sender terminal (communication terminal10A) recognize the current operating state of each of the communicationterminals identified by the communication IDs (“01b1”, “01b2”, and“01b3”), which are candidates for the counterpart terminal with whichthe request sender terminal (the communication terminal 10A) is tocommunicate. The communication terminal 10A displays a destinationselection screen as illustrated in FIG. 16.

Further, the terminal extraction unit 54 of the communication managementsystem 50 searches the destination list management table 701 by usingthe communication ID “01aa” of the request sender terminal (thecommunication terminal 10A) from which the login request is received asa search key. By this searching, the terminal extraction unit 54extracts the communication ID(s) of one or more other request senderterminals that have registered the communication ID “01aa” of therequest sender terminal (the communication terminal 10A) as acounterpart terminal candidate (step S29). In case of the destinationlist management table 701 of Table 3, the extracted communication IDs ofthe other request sender terminals are “01b1”, “01b2” and “01b3”.

Next, the terminal management unit 53 of the communication managementsystem 50 searches the terminal management table 603 by using thecommunication ID “01aa” of the request sender terminal (thecommunication terminal 10A) from which the login request is received asa search key. By this searching, the terminal management unit 53acquires the operating state of the request sender terminal (thecommunication terminal 10A) from which the login request is received(step S30).

Next, the terminal management unit 53 of the communication managementsystem 50 selects, from among the communication IDs (“01b1”, “01b2”,“01b3”) extracted in step S29, one or more communication IDs (“01b1”,“01b2”, “01b3”) associated with the operating state of “Online(available for communication)” in the terminal management table 603.

Further, the data exchange unit 51 transmits counterpart stateinformation including the communication ID “01aa” of the request senderterminal (communication terminal 10A) and the operating state “Online(available for communication)” to the communication terminal 10B1corresponding to the extracted communication IDs (“01b1”, “01b2”,“01b3”) (step S31).

On the other hand, the other communication terminal 10B1 also performsthe same processes as those in steps S22 to S32, in response to anoperation of turning on the power, for example. For example, anadministrator of the mobile apparatus 20, for example, turns on thecommunication terminal 10B1.

<Destination Selection Screen>

FIG. 16 is an illustration of an example of the destination selectionscreen displayed on the communication terminal 10A. On the destinationselection screen 1201 illustrated in FIG. 16, a message 1202 for guidingthe operator to select a desired communication terminal 10 as acounterpart and a plurality of buttons 1203 corresponding to theplurality of communication terminals 10 as counterpart terminals. Theoperator of the communication terminal 10A selects one of the displayedplurality of buttons 1203, to select a counterpart terminal to which arequest for participation in a session is to be sent. In other words,the operator selects the mobile apparatus 20 to be controlled. Asillustrated in FIG. 16, since the operator can select a plurality ofcommunication terminals 10B, the operator can recognize the situation ofdifferent sites from the same place.

<Communication Processes>

FIG. 17A and FIG. 17B are a sequence diagram illustrating an example ofcommunication processes performed by the communication system 1. In thepresent embodiment, a description is given of an example of acommunication management for starting communication between thecommunication terminal 10A and the communication terminal 10B1, which isa device control terminal configured to control the mobile apparatus 20.

In step S901, the operation input receiving unit 12 of the communicationterminal 10A receives a selection operation of a counterpart terminal(the communication terminal 10B1) by the operator of the communicationterminal 10A.

The data exchange unit 11 of the communication terminal 10A transmitsstart request information for requesting the start of a session to thecommunication management system 50 (step S902). This start requestinformation includes, for example, a communication ID of a requestsender terminal (i.e., the communication ID of the communicationterminal 10A as a request sender terminal), and a communication ID of acounterpart terminal (i.e., the communication ID of the communicationterminal 10B1 as the counterpart terminal). The start requestinformation further includes information such as an IP address (requestsource IP address) of the communication terminal 10A.

In step S903, in response to receiving the start request informationfrom the communication terminal 10A, the terminal management unit 53 ofthe communication management system 50 updates the terminal managementDB 5003 based on the communication ID “01aa” of the request senderterminal (the communication terminal 10A) included in the start requestinformation. For example, the terminal management unit 53 changesinformation of the operating state corresponding to the communication ID“01aa” of the communication terminal 10A to “Online (communicating)” andupdates information of the reception date and time.

In step S904, the session management unit 55 of the communicationmanagement system 50 transmits start request information for requestingthe start of a session to the communication terminal 10B1, which is acounterpart terminal. This start request information includes, forexample, the communication ID of the request sender terminal (i.e., thecommunication terminal 10A as a request sender terminal).

In step S905, in response to receiving the start request informationfrom the communication management system 50, the communication terminal10B1 transmits start response information to the communicationmanagement system 50. This start response information includes, forexample, the communication ID of the counterpart terminal (e.g., thecommunication terminal 10B1). In the present embodiment, the startresponse information is transmitted without any operation on thecommunication terminal 10B1 side. In another example, the communicationterminal 10B1 transmits the start response information in response to anoperation by the administrator.

In step S906, in response to receiving the start response informationfrom the communication terminal 10B1, the terminal management unit 53 ofthe communication management system 50 updates the terminal managementDB 5003 based on the communication ID “01b1” of the communicationterminal 10B1 included in the start response information. For example,the terminal management unit 53 changes information of the operatingstate corresponding to the communication ID “01b1” of the communicationterminal 10B1 to “Online (communicating)”, and updates information ofthe reception date and time.

In step S907, the session management unit 55 of the communicationmanagement system 50 generates a session ID, which is identificationinformation for identifying a session. Further, the session managementunit 55 stores, in the session management DB 5005, the generated sessionID in association with the communication ID of the request senderterminal (the communication ID of the communication terminal 10A) andthe communication ID of the counterpart terminal (the communication IDof the communication terminal 10B1).

In step S909, the session management unit 55 of the communicationmanagement system 50 transmits session information to the relayapparatus 30. This session information includes, for example,information such as the session ID generated in step S907. The relayapparatus 30 can acquire the session information from the sessionmanagement DB 5005 based on the session ID.

In step S910 a, the session management unit 55 of the communicationmanagement system 50 transmits start instruction information forinstructing the communication terminal 10A to start a session. Insubstantially the same manner, in step S910 b, the session managementunit 55 of the communication management system 50 transmits startinstruction information for instructing the communication terminal 10B1to start a session. The start instruction information includes thesession ED, and the communication terminal 10 can acquire sessioninformation from the session management DB 5005 based on the session ID.

In step S911 a, the communication terminal 10A establishes a sessionwith the relay apparatus 30 based on the received start instructioninformation. In substantially the same manner, in step 911 b, thecommunication terminal 10B1 establishes a session with the relayapparatus 30 based on the received start instruction information. Thus,the communication terminal 10A and the communication terminal 10B1 canparticipate in the same session.

In step S911 c, the wide-angle image transmitting unit 31 of thewide-angle image capturing apparatus 9 transmits an equirectangularprojection video to the communication terminal 10B1. For example, whenthe communication terminal 10B1 establishes a session, the communicationterminal 10B1 instructs the wide-angle image capturing apparatus 9 startimaging and transmission of an equirectangular projection video. Inanother example, the wide-angle image capturing apparatus 9 cancontinuously transmit an equirectangular projection video to thecommunication terminal 10B1 during a time period when the power of thewide-angle image capturing apparatus 9 is on. In still another example,an equirectangular projection video can be transmitted beforeestablishment of a session.

The communication terminal 10A and the communication terminal 10B1participate in a session identified by the same session ID, and thenexchange content data such as image data and audio data with each other,to perform a videoconference, for example (S912). This image dataincludes an equirectangular projection video, a front-side video, and arear-side video.

Further, the communication terminal 10A can exchange informationrelating to an operation of a device (e.g., the mobile apparatus 20)with the communication terminal 10B1 by using the established session.

In another example, the information relating to the operation of thedevice can be exchanged by using a control session via the communicationmanagement system 50, instead of by using the session. In still anotherexample, the information relating the operation of the device can beexchanged between the communication terminal 10A and the communicationterminal 10B1 via the communication network 2 or the like.

In the present embodiment, the following description is provided on theassumption that the communication terminal 10A exchanges the informationrelating to the operation of the device (e.g., the mobile apparatus 20)with the communication terminal 10B1 by using the established session.

In step S913, in response to an input by the operator of thecommunication terminal 10A on a screen that accepts instructions foroperating the device, the communication terminal 10A transmits operationinstruction information corresponding to the input is transmitted byusing the session. This operation instruction information includes, forexample, the communication ID of the communication terminal 10A and anoperation instruction corresponding to the operation content by theoperator.

In step S914, the device control unit 19 c of the communication terminal10B1 controls the mobile apparatus 20 via the inter-device communicationunit 19 d based on the operation instruction included in the operationinstruction information transmitted from the communication terminal 10A.

<Pattern 1>

A description is now given of examples of the video layout patterns 1 to3. FIG. 18 is an illustration of an example of a video display screen610 displayed on the display 109 by the communication terminal 10A inPattern 1. The video display screen 610 includes a video display area611, a front-side video button 612, a rear-side video button 613, andthe mobile apparatus operation button 614. In the video display area 611of an illustration (a) of FIG. 18, a spherical video is displayed. Thedisplay control unit 16 of the communication terminal 10A maps theequirectangular projection video transmitted from the communicationterminal 10B to the surface of a solid sphere to generate the sphericalvideo (see FIG. 10A and FIG. 10B).

The operator can rotate the spherical video horizontally and verticallywith a finger, a mouse or the like. Further, the operator can reduce thespherical video by pinching in on the spherical video with a finger, andcan enlarge the spherical video by pinching out. Thus, the communicationterminal 10A can display a desired predetermined-area image Q in thevideo display area 611 according to an operator's operation.

The front-side video button 612 is a button for switching a video to bedisplayed in the video display area 611 to a front-side video. Therear-side video button 613 is a button for switching a video to bedisplayed in the video display area 611 to a rear-side video. The mobileapparatus operation button 614 has four buttons. The four buttonscorrespond to buttons for controlling the mobile apparatus 20 to moveback, forth, right and left, respectively. Since the mobile apparatusoperation button 614 is displayed as being superimposed on the sphericalvideo, the display area is larger than a display area of a case wherethe mobile apparatus operation button 614 is displayed independently.This makes the display area suitable for viewing.

An illustration (b) of FIG. 18 is an illustration of the video displayscreen 610 in which a front-side video is displayed in the video displayarea 611 in response to pressing of the front-side video button 612.Since the front-side video button 612 is pressed, the front-side videois displayed in the video display area 611. In addition, the sphericalvideo button 615 is displayed instead of the front-side video button612. The spherical video button 615 is a button for switching a video tobe displayed in the video display area 611 to a spherical video.

Since the mobile apparatus operation button 614 is displayed as beingsuperimposed on the front-side video or the rear-side video, the displayarea is larger than a display area of a case where the mobile apparatusoperation button 614 is displayed independently. This makes the displayarea suitable for viewing.

An illustration (c) of FIG. 18 is an illustration of the video displayscreen 610 in which a rear-side video is displayed in the video displayarea 611 in response to pressing of the rear-side video button 613. Thespherical video button 615 and the front-side video button 612 aredisplayed instead of the rear-side video button 613.

As described above, by pressing the front-side video button 612, therear-side video button 613, and the spherical video button 615, a videoto be displayed in the video display area 611 is switched from one toanother.

FIG. 19 is a flowchart illustrating an example of an operation ofswitching a video to be displayed in the video layout example of Pattern1. The operation illustrated in FIG. 19 starts when the communicationterminal 10A receives at least one of a spherical video, a front-sidevideo, and a rear-side video.

First, the display control unit 16 controls the display 109 to display aspherical video (S101). In this case, the mobile apparatus operationbutton 614 is displayed as being superimposed on the spherical video.When the spherical video is displayed, the operation input receivingunit 12 receives an operation for rotation, enlargement, and/orreduction. The display control unit 16 displays an image of thepredetermined area T according to the received operation in the videodisplay area 611.

Displaying the spherical video first as described above is just oneexample. In another example, a front-side video or a rear-side video isdisplayed first. In still another example, any one of the sphericalvideo, the front-side video and the rear-side video is displayed in anorder that the communication terminal 10A receives these videos.

The display control unit 16 controls the display 109 to display thefront-side video button 612 and the rear-side video button 613 (S102).

The operation input receiving unit 12 determines whether the front-sidevideo button 612 is pressed (S103). When the operation input receivingunit 12 determines that the front-side video button 612 is not pressed(S103: NO), the operation proceeds to step S106.

When the operation input receiving unit 12 determines that thefront-side video button 612 is pressed (S103: YES), the display controlunit 16 displays a front-side video (S104). The display control unit 16controls the display 109 to display the rear-side video button 613 andthe spherical video button 615 (S105).

The operation input receiving unit 12 determines whether the rear-sidevideo button 613 is pressed (S106). When the operation input receivingunit 12 determines that the rear-side video button 613 is not pressed(S106: NO), the operation proceeds to step S109.

When the operation input receiving unit 12 determines that the rear-sidevideo button 613 is pressed (S106: YES), the display control unit 16displays a rear-side video (S107). The display control unit 16 controlsthe display 109 to display the front-side video button 612 and thespherical video button 615 (S108).

The operation input receiving unit 12 determines whether the sphericalvideo button 615 is pressed (S109). When the operation input receivingunit 12 determines that the spherical video button 615 is pressed (S109:YES), the operation proceeds to step S101. When the operation inputreceiving unit 12 determines that the spherical video button 615 is notpressed (S109: NO), the operation proceeds to step S103.

As described above, the communication terminal 10A can video in Pattern1 in response to receiving the pressing of the front-side video button612, the rear-side video button 613, or the spherical video button 615.

As described above, in the video layout example of Pattern 1, a video tobe displayed on one display 109 is switched from among the sphericalvideo, the front-side video and the rear-side video. This secures thesizes of the spherical video and the front-side video (rear-side video).Such layout is suitable for viewing by the operator. In a state wherethe spherical video is displayed, the operator can check 360-degreesurroundings around the wide-angle image capturing apparatus 9 (i.e. thecommunication terminal 10B) by rotating the spherical video. Inaddition, since the front-side video has a high resolution, displayingthe front-side video is suitable when the operator wants to readcharacters on a signboard or the like or when the operator wants tocheck the structure of an object.

<Pattern 2>

FIG. 20 is an illustration of an example of a video display screen 610displayed on the display 109 by the communication terminal 10A inPattern 2. The video display screen 610 in the illustration (a) of FIG.20 includes an entire view area 621, a front-side video area 622, andthe mobile apparatus operation button 614. In the entire view area 621,an equirectangular projection video is displayed. For example, theequirectangular projection video in the illustration (a) of FIG. 20 isvideo obtained by imaging a room in a model house, and the videoincludes four walls and some furniture. As illustrated, in theequirectangular projection video, objects in 360-degree surroundings isrepresented on a plane.

In the front-side video area 622, a front-side video is displayed. Inthe front-side video in the illustration (a) of FIG. 20, a shelf 622 ais displayed. The equirectangular projection video is greatly distortedcompared to the front-side video, which has little distortion and anarrower angle of view than the equirectangular projection video.Further, the front-side video is displayed larger than theequirectangular projection video. Since the resolution of the front-sidevideo is higher than the resolution of the equirectangular projectionvideo, the operator is likely to obtain detailed information from thefront-side video. For example, the front-side video is suitable when theoperator wants to read characters on a signboard or the like or when theoperator wants to check the structure of a product. Furthermore,although the equirectangular projection video distorted and displayedsmaller than the front-side video, the equirectangular projection videois always displayed on the video display screen 610. This allows theoperator to recognize the situation around the mobile apparatus 20.

Since the mobile apparatus operation button 614 is displayed as beingsuperimposed on the front-side video, the display area is larger than adisplay area of a case where the mobile apparatus operation button 614is displayed independently. This makes the display area suitable forviewing.

The front-side video area 622 includes the rear-side video button 613that allows the user to switch the front-side video to a rear-sidevideo.

An illustration (b) of FIG. 20 is an illustration of an example of thevideo display screen 610 displayed when the rear-side video button 613is pressed. The video display screen 610 in the illustration (b) of FIG.20 has a rear-side video area 623 instead of the front-side video area622. In the rear-side video area 623, a rear-side video is displayed.The rear-side video area 623 includes the front-side video button 612that allows the user to switch the rear-side video to the front-sidevideo.

In addition, when the videos are displayed as in the illustrations (a)and (b) of FIG. 20, the operate can perform operation for arranging thefront-side video or the rear-side video on the upper side and theequirectangular projection video on the lower side. For example, whenthe user swipes (or flick) the entire view area 621 downward, or swipes(or flick) the front-side video area 622 or the rear-side video area 623upward, the communication terminal 10A displays the video display screen610 as in the illustrations (c) and (d) of FIG. 20.

In the illustration (c) of FIG. 20, the front-side video is displayed onthe upper side, and the equirectangular projection video is displayed onthe lower side. In the illustration (d) of FIG. 20, the rear-side videois displayed on the upper side, and the equirectangular projection videois displayed on the lower side. As described, the user can control thecommunication terminal 10A to display the front-side video, therear-side video, and the equirectangular projection video in the user'sdesired arrangement.

FIG. 21 is a flowchart illustrating an example of an operation ofdisplaying videos in the video layout example of Pattern 2. Theoperation illustrated in FIG. 21 starts when the communication terminal10A receives at least one of a spherical video, a front-side video, anda rear-side video.

The display control unit 16 displays an equirectangular projection videoin the entire view area 621 (S201).

The display control unit 16 displays a front-side video in thefront-side video area 622 (S202). In this case, the mobile apparatusoperation button 614 is displayed as being superimposed on the sphericalvideo. In another example, the rear-side video is displayed first. Instill another example, the equirectangular video, the front-side videoand the rear-side video is displayed in an order that the communicationterminal 10A receives these videos.

Next, the operation input receiving unit 12 determines whether therear-side video button 613 is pressed (S203). When the operation inputreceiving unit 12 determines that the rear-side video button 613 ispressed (S203: YES), the display control unit 16 displays a rear-sidevideo in the rear-side video area 623 (S204).

Next, the operation input receiving unit 12 determines whether anup-and-down switching operation such as swiping is input (S205). Whenthe operation input receiving unit 12 determines the up-and-downswitching operation is input (S205: YES), the display control unit 16switches the display positions of the front-side video area 622 and theentire view area 621 or the display positions of the rear-side videoarea 623 and the entire view area 621 in an up-and-down directions(S206).

Thus, the equirectangular projection video and the front-side video, orthe equirectangular projection video and the rear-side video aredisplayed together on one video display screen 610.

A description is now given of another display example of Pattern 2 withreference to FIG. 22. FIG. 22 is an illustration of an example of thevideo display screen 610 displayed on the display 109 by thecommunication terminal 10A in Pattern 2. The video display screen 610 inan illustration (a) of FIG. 22 includes a spherical video area 631, thefront-side video area 622, and the mobile apparatus operation button614. In the spherical video area 631, a spherical video is displayed.The display control unit 16 of the communication terminal 10A maps anequirectangular projection video transmitted from the communicationterminal 10B to the surface of a solid sphere to generate the sphericalvideo (see FIG. 10A and FIG. 10B). In substantially the same manner asdescribed with reference with the illustrations (b) to (d) of FIG. 20,the illustrations (b) to (d) of FIG. 22 represent states in which thefront-side video and the rear-side video are switched, the displaypositions of the spherical video and the front-side video are switched,or the display positions of the spherical video and the rear-side videoare switched.

Although in the illustration (a) of FIG. 22, an image of a copier 631 ais included in the spherical video area 631, this is for the sake ofexplanatory convenience. The communication terminal 10A can display adesired predetermined-area image Q in the spherical video area 631according to an operator's operation. The same description of thefront-side video area 622 provided above with reference to theillustration (a) of FIG. 20 applied to the front-side video area 622illustrated in FIG. 22. In another example, the mobile apparatusoperation button 614 can be displayed in the spherical video area 631.

In the illustration (a) of FIG. 22, the spherical video area 631 and thefront-side video area 622 have substantially the same size. An image(video) of a desired direction that the operator wants to view can bedisplayed in a large size in the spherical video area 631. Although thefront-side video area 622 is smaller compared to that of theillustration (a) of FIG. 20, since the front-side video has a highresolution, the operator can obtain detailed information from thefront-side video. Accordingly, since the spherical video area 631 andthe front-side video area 622 have the same size, the advantages of aspherical video and a front-side video can be provided.

FIG. 23 is a flowchart illustrating an example of an operation ofdisplaying videos in the another display example of Pattern 2illustrated in FIG. 22. The processes in the operation illustrated inFIG. 23 are performed in substantially the same manner as describedabove with reference to FIG. 21, except that an equirectangularprojection video is replaced with a spherical video. With the operationof FIG. 23, a spherical video and a front-side video, or a sphericalvideo and a rear-side video are displayed together on one video displayscreen 610.

Although in Pattern 1 and Pattern 2, the orientation of the screen ofthe communication terminal 10A is portrait (vertical), this is just oneexample. Alternatively, the orientation of the screen of thecommunication terminal 10A can be landscape (horizontal). In this case,it is preferable to maintain the aspect ratio of the videos.Alternatively, the videos can be enlarged or reduced in the vertical orhorizontal direction as the aspect ratio being changed.

<Pattern 3>

FIG. 24 is an illustration of an example of the video display screen 610displayed on the display 109 by the communication terminal 10A inPattern 3. The video display screen 610 illustrated in FIG. 24 includesthe spherical video area 631, a sub front-side video area 642, a subrear-side video area 643, and the mobile apparatus operation button 614.In the spherical video area 631, a spherical video is displayed. In subfront-side video area 642, a front-side video is displayed. In the subrear-side video area 643, a rear-side video is displayed. An objectincluded in the front-side video displayed in the sub front-side videoarea 642 is present in a direction that is opposite by 180 degreesviewed from the mobile apparatus 20 to a direction in which anotherobject included in the back video displayed in the sub rear-side videoarea 643.

In the video display screen 610 of FIG. 24, the spherical video area 631is larger than any of the sub front-side video area 642 and the subrear-side video area 643. Therefore, the spherical video is a mainvideo, and each of the front-side video and the rear-side video is a subvideo. An image (video) of a desired direction that the operator wantsto view can be displayed in a large size in the spherical video area631. In addition, since the front-side video and the rear-side video arealways displayed, the operator can always recognize the travelingdirection of the mobile apparatus 20. Further, the operator can alwaysrecognize the situation behind the travelling direction.

Since the mobile apparatus operation button 614 as being superimposed onthe spherical video, the display area is larger than a display area of acase where the mobile apparatus operation button 614 is displayedindependently. This makes the display area suitable for viewing.

Although in the above description, the spherical video area 631 islarger than any of the sub front-side video area 642 and the subrear-side video area 643, this is just one example. In another example,the front-side video is displayed larger than the spherical video andthe rear-side video. In still another example, the rear-side video isdisplayed larger than the spherical video and the front-side video.Further, it is preferable that the operator can configure a desiredsetting as to which video is to be displayed in each of the threedisplay areas (the spherical video area 631, the sub front-side videoarea 642, and the sub rear-side video area 643).

FIG. 25 is a flowchart illustrating an example of an operation ofdisplaying videos in the video layout example of Pattern 3. Theoperation illustrated in FIG. 25 starts when the communication terminal10A receives at least one of a spherical video, a front-side video, anda rear-side video.

The display control unit 16 displays a spherical video in the sphericalvideo area 631 (S301). In this case, the mobile apparatus operationbutton 614 is displayed as being superimposed on the spherical video.

The display control unit 16 displays a front-side video in the subfront-side video area 642 (S302). The display control unit 16 displays arear-side video in the sub rear-side video area 643 (S303).

Thus, the spherical video, the front-side video and the rear-side videoare displayed together on one video display screen 610.

A description is now given of another display example of Pattern 3 withreference to FIG. 26. FIG. 26 is an illustration of an example of thevideo display screen 610 displayed on the display 109 by thecommunication terminal 10A in Pattern 3. The video display screen 610illustrated in FIG. 26 includes a main display area 651, one or morevarious video areas 652 (various video areas 652 a to 652 c), and themobile apparatus operation button 614. In the main display area 651, aspherical video is displayed. The display control unit 16 of thecommunication terminal 10A maps an equirectangular projection videotransmitted from the communication terminal 10B to the surface of asolid sphere to generate the spherical video (see FIG. 10A and FIG.10B).

In the one or more various video areas 652, a desired predetermined-areaimage Q of the spherical video can be displayed as well as thefront-side video and the rear-side video.

The main display area 651 occupies almost the entire area of the display109, and the one or more various video areas 652 is displayed movably asbeing superimposed on the main display area 651. In an exampleillustrated in FIG. 26, three various video areas 652 a to 652 c aredisplayed. Each of the various video areas 652 is a window. The operatorcan change the position and/or the size of one or more of the variousvideo areas 652, delete (hide, close the window) one or more of thevarious video areas 652, minimize one or more of the various video areas652, and generate a new window as one of the various video areas 652.

When displaying videos in Pattern 3, the display control unit 16displays a front-side video and a rear-side video in the various videoareas 652. Further, the display control unit 16 displays one or morepredetermined-area images Q obtained by duplicating arbitrary one ormore partial areas of the spherical video in the various video areas652. Since the operator can change the predetermined-area image Q, animage of any area of the spherical image can be displayed by default asthe predetermined-area image Q in one or more of the various video areas652. However, since the front-side video and the rear-side video aredisplayed in the various video areas 652, it is preferable to duplicatean area other than areas corresponding to the front-side video and therear-side video as the predetermined-area image Q. For example, imagesof the right side and the left side in the horizontal direction withrespect to the front of the mobile apparatus 20 are preferable as thepredetermined-area image Q displayed in one or more of the various videoareas 652.

The default positions of the various video areas 652 are, for example,the four corners of the main display area 651. The operator can move thevarious video areas 652 to any desired positions.

In view of the sizes of the main display area 651 and the various videoareas 652, a video displayed in the main display area 651 is a mainvideo, and videos displayed in the various video areas 652 are subvideos. The operator can view the spherical video on a large screen, andcan cause the communication terminal 10A to display an image of adirection that the operator wants to view in a large size. Further,since the front-side video and the rear-side video are displayed in thevarious video areas 652, the operator can recognize the situation in thetraveling direction of the mobile apparatus 20 and the situation in theopposite direction of the travelling direction. In addition, when thespherical video(s) are displayed in one or more of the various videoareas 652, the operator can recognize the situation of a desireddirection around the mobile apparatus 20. The spherical video displayedin one of the various video areas 652 is an example of a duplicatevideo. When the operator performs an operation of generating a newwindow of the various video areas 652, the operation input receivingunit 12 receives this operation, and the display control unit 16 newlydisplays the spherical video in the generated window of the variousvideo areas 652.

The operator can perform operation for moving one or more of the variousvideo areas 652 to a desired position such that the one or more of thevarious video areas 652 blocks the operator's view. In addition, theoperator can perform an operation for deleting or minimizing one or moreof the various video areas 652. The operation input receiving unit 12receives such operator's operation, and for example, the display controlunit 16 hides or minimizes one or more of the various video areas 652according to the received operation.

Conventionally, as many image capturing devices as the number of variousvideo areas 652 were needed. In the present embodiment, by thedisplaying processes as described above, one or more arbitrarypredetermined-area images Q obtained by duplicating one or morearbitrary partial area of the spherical video are displayed. Thisenables to always display videos of many directions at low cost.

Since the mobile apparatus operation button 614 is displayed as beingsuperimposed on the spherical video, the display area is larger than adisplay area of a case where the mobile apparatus operation button 614is displayed independently. This makes the display area suitable forviewing.

FIG. 27 is a flowchart illustrating an example of an operation ofdisplaying videos in the video layout example of Pattern 3 describedwith reference to FIG. 26. The operation illustrated in FIG. 27 startswhen the communication terminal 10A receives at least one of a sphericalvideo, a front-side video, and a rear-side video.

The display control unit 16 displays a spherical video in the maindisplay area 651 (S401). In this case, the mobile apparatus operationbutton 614 is displayed as being superimposed on the spherical video.

The display control unit 16 displays a front-side video as beingsuperimposed on the spherical video (S402). The display control unit 16displays a rear-side video as being superimposed on the spherical video(S403). For example, the initial positions of the various video areas652 in which the front-side video and the rear-side video are displayeddetermined in advance. Alternatively, the initial positions of thevarious video areas 652 in which the front-side video and the rear-sidevideo are displayed are the positions that were set when the userdisplayed the video display screen 610 of FIG. 26 last time.

The operation input receiving unit 12 determines whether a new window asone of the various video areas 652 is added (S404). An operation foradding a new window as one of the various video areas 652 can be anyoperation. For example, the operator performs a long tap or a double tapto cause the communication terminal 10A to display a menu, and selects(presses) a message “Add various video areas” included in the menu. Inaddition, although the menu can include selection buttons for selectinga front-side video and a rear-side video, when a front-side video and/ora rear-side video is already displayed, it is preferable that theselection button for selecting a front-side video and/or a rear-sidevideo is not displayed. Thus, one or less front-side video and rear-sidevideo can be displayed together with a spherical video.

When a new window as one of the various video areas 652 is added (S404:YES), the display control unit 16 sets a predetermined-area image Q of apredetermined area T in the spherical video corresponding to a positionwhere the operator performs a long tap or a double tap to the new windowas one of the various video areas 652, and displays the new windowincluding the predetermined-area image Q as being superimposed on thespherical video (S405).

Patterns 1 to 3 are desired heretofore. The operator can switch thedisplay from among Patterns 1 to 3 at will by a predetermined operationsuch as pressing a button.

<Switching Patterns>

The operator can switch Patterns 1 to 3 from one to another at will. Inaddition, if Patterns 1 to 3 are automatically switched from one toanother in response to some event, the usability of the operator isimproved.

A description is now given of triggers for switching the pattern withreference to FIG. 28A and FIG. 28B. FIG. 28A and FIG. 28B are flowchartsfor explaining examples of triggers for automatically switching thepatterns.

A description is now given of an operation illustrated in the flowchartof FIG. 28A. The operation illustrated in the flowchart of FIG. 28A isstarted when the communication terminal 10A establishes a session.

In response to the establishment of a session, the display control unit16 of the communication terminal 10A can display a spherical video, afront-side video, and a rear-side video. First, the display control unit16 displays a spherical video in the video display area 611 in Pattern 1(S1001). In other words, the spherical video is displayed on the entireare of the display 109.

Next, the display control unit 16 rotates the spherical video in thehorizontal direction at a constant speed (S1002). The process of stepS1002 is implement by changing the azimuth angle aa at a constant speedwith the elevation angle ea constant in the horizontal direction whilekeeping the angle of view α constant. As a result, to the operator, thepredetermined-area image Q appears to rotate.

The display control unit 16 determines whether the spherical video hasbeen rotated by 360 degrees in the horizontal direction (S1003). Thisdetermination is made by storing the direction (ea, aa) of the centerpoint CP before starting the rotation. The rotation continues until thespherical video rotates by 360 degrees in the horizontal direction.

When the display control unit 16 determines that the spherical videorotates by 360 degrees in the horizontal direction (S1003: YES), thedisplay control unit 16 switches the current Pattern 1 to Pattern 2 orPattern 3 (S1004).

Immediately after the session is established, the operator does not knowthe surrounding situation. With the processes as described withreference to FIG. 28A, the predetermined-area image Q is rotated by 360degrees immediately after the session is established, and then videos orimages are displayed in Patterns 2 or 3. This allows the operator tofirst recognize the situation, and then check the situation of the frontby viewing a front-side video and start moving. Alternatively, thefront-side video can be displayed large with the display patternunchanged from Pattern 1.

A description is now given of an operation illustrated in the flowchartof FIG. 28B. The operation illustrated in the flowchart of FIG. 28A isstarted when the communication terminal 10A establishes a session.

In response to the establishment of a session, the display control unit16 of the communication terminal 10A can display a spherical video, afront-side video, and a rear-side video. First, the display control unit16 displays an equirectangular projection video and a front-side videoin Pattern 2 (S2001). In other words, the equirectangular projectionvideo is displayed small in the upper part of the display 109, and thefront-side video is displayed in the lower part of the display 109.

The display control unit 16 determines whether a predetermined timeperiod has elapsed since the videos are displayed in Pattern 2 (S2002).The display control unit 16 keeps displaying the videos in Pattern 2until the predetermined time period passes.

When the display control unit 16 determines that the predetermined timeperiod has elapsed (S2002: YES), the display control unit 16 switchesthe current Pattern 2 to Pattern 1 or Pattern 3 (S2003).

Immediately after the session is established, the operator does not knowthe surrounding situation. With the processes as described withreference to FIG. 28B, the equirectangular projection video is firstdisplayed in Pattern 2 for a predetermined time period after the sessionis established, and then Pattern 2 is switched to Pattern 1 or Pattern3. This allows the operator to first recognize the situation, and thencheck the situation of the front by viewing a front-side video and startmoving. Alternatively, the display of Pattern 2 can be continued.

As described heretofore, the communication terminal 10A according to thepresent embodiment appropriately arranges and displays a spherical videoand a front-side video (rear-side video). This allows the operator ofthe communication terminal 10A to check an overview situation around themobile apparatus 20 or check the front-side video or the rear-side videohaving high resolution. Displaying the spherical video and thefront-side video (rear-side video) in the layouts of Patterns 1 to 3allows the operator to cause the mobile apparatus 20 to move by usingthe spherical video and the front-side video (rear-side video) or tocheck the surroundings. Also, Patterns 1 to 3 can be switched from oneto another at the operator's will or automatically.

<Variation>

The above-described embodiments are illustrative and do not limit thepresent disclosure. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present disclosure. Any one of the above-describedoperations may be performed in various other ways, for example, in anorder different from the one described above.

For example, in the above description of the present embodiment, thewide-angle image capturing apparatus 9 transmits an equirectangularprojection video to the communication terminal 10B, and thecommunication terminal 10B transmits the equirectangular projectionvideo to the communication terminal 10A. Alternatively, the wide-angleimage capturing apparatus 9 can directly establish a session with thecommunication terminal 10A to transmit an equirectangular projectionvideo. In this case, the wide-angle image capturing apparatus 9 is oneform of the communication terminal 10. In other words, the wide-angleimage capturing apparatus 9 is the communication terminal 10 thattransmits video but does not receive video.

Further, although in the above description of the present embodiment,the communication terminal 10B transmits a rear-side video, thecommunication terminal 10B may not transmit a rear-side video. Inanother example, the communication terminal 10B can transmit an imagehaving a normal angle of view in addition to or in alternative to afront-side video and a rear-side video.

Further, Patterns 1 to 3 are not independent from one another, and onecommunication terminal 10 can change the display pattern such as fromPattern 1 to Pattern 2, from Pattern 2 to Pattern 3, and Pattern 3 to 1(or with the order reversed).

Further, in the above description of the present embodiment, the mobileapparatus 20 is an apparatus that moves on land. In another example, themobile apparatus 20 can be an apparatus that flies in the air such as adrone. In still another example, the mobile apparatus 20 can be anapparatus that travels on the sea or moves in the water. In stillanother example, the mobile apparatus 20 can be an apparatus that moveson an underground passageway, on a narrow road, or underground. Inaddition, when the mobile apparatus 20 is an apparatus that moves onland, the mobile apparatus 20 can move with wheels, with multiple legssuch as two, three or four legs, or with Caterpillar (registeredtrademark).

Further, the display method of the present embodiment can be applied notonly to a telepresence robot but images (video) captured by thewide-angle image capturing apparatus 9 mounted to an apparatus moved bya person and images (video) captured by a planar image capturingapparatus.

Further, although in the present embodiment, a description is givenabove of an example case in which one operator operates one mobileapparatus 20, one operator can operate a plurality of mobile apparatuses20. In this case, the communication terminal 10A establishes a sessionwith a plurality of communication terminals 10B. In another example,multiple operators can operate one mobile apparatus 20. In this case,for example, an operation instruction information transmitted to thecommunication terminal 10B last is a valid operation instructioninformation. Alternatively, the concept of an operation right can beintroduced so that only the communication terminal 10A having theoperation right can transmit an operation instruction. The operationright can be transferred by an operation by the operator, for example.

In the above description of the present embodiment, operationinstruction information is transmitted in a communication session.Alternatively, operation instruction information can be transmitted in asession of content data.

Further, in the above description of the present embodiment, thecommunication terminals 10 communicates with each other via the relayapparatus 30. Alternatively, the communication terminals 10 cancommunicate with each other without intervention of the relay apparatus30. In such communication, a communication protocol such as webreal-time communication (WebRTC) is used.

In addition, in the example of a functional configuration as illustratedin FIG. 14A and FIG. 14B, processes performed by the communicationmanagement system 50 and the communication terminal 10 are divided basedon main functions in order to facilitate understanding of processes. Nolimitation is intended by how the processes are divided or by the nameof the processes. The processes implemented by communication managementsystem 50 and the communication terminal 10 can be divided to a largernumber of processes according to the contents of processes. Further, oneprocess can be divided to include a larger number of processes.

In the present embodiment, a description is given of an example case inwhich the communication management system 50 and the relay apparatus 30are independent apparatuses for the sake of explanatory convenience. Inanother example, an apparatus in which the functions of thecommunication management system 50 and the relay apparatus 30 canprovide the functions of the communication management system 50 and therelay apparatus 30.

In still another example, the communication system 1 can include aplurality of communication management systems 50, and the functions ofthe communication management system 50 can be distributed over theplurality of servers.

In still another example, one or more of the databases that thecommunication management system 50 includes in the storage unit 5000 canbe present on the communication network 2.

The data exchange unit 11 is an example of communication means. Thedisplay control unit 16 is an example of display processing means. Theoperation input receiving unit 12 is an example of receiving means.

In the related art, a controller for controlling a mobile apparatus isnot considered. In other words, there is no disclosure on how to displaythe controller while displaying a plurality of videos transmitted fromthe mobile apparatus.

According to one or more embodiments of the present disclosure, acommunication terminal is provided that can control a robot while anoperator is looking at images that are captured by of a plurality ofimage capturing apparatuses and transmitted from a mobile apparatus.

According to one or more embodiments, a communication terminal isprovided that displays images suitable for viewing to an operator, theimages being captured by a plurality of image capturing apparatuses andtransmitted from a mobile apparatus.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), digital signal processor (DSP), fieldprogrammable gate array (FPGA), and conventional circuit componentsarranged to perform the recited functions.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention.

Any one of the above-described operations may be performed in variousother ways, for example, in an order different from the one describedabove.

What is claimed is:
 1. A communication terminal, communicable withanother communication terminal mounted on a mobile apparatus or with acommunication device of the mobile apparatus via a network, fortransmitting operation instruction information for controlling themobile apparatus to the another communication terminal or thecommunication device, the communication terminal comprising circuitryconfigured to: receive a first video transmitted by one of the anothercommunication terminal and the communication device and a second videohaving a wide-angle of view captured by a wide-angle image capturingapparatus, the wide-angle image capturing apparatus being connected toone of the mobile apparatus and the another communication terminal orbeing configured as a single unit with one of the mobile apparatus andthe another communication terminal; display, on a display device, atleast one of the first video and the second video that are received; anddisplay, on the display device, a controller for controlling anoperation of the mobile apparatus as being superimposed on the at leastone of the first video and the second video.
 2. The communicationterminal of claim 1, wherein the circuitry is further configured todisplay a display area of the first video on the display device in alarger size compared to a display area of the second video.
 3. Thecommunication terminal of claim 1, wherein the circuitry is furtherconfigured to display a display area of the second video on the displaydevice in a larger size compared to a display area of the first video.4. The communication terminal of claim 1, wherein the circuitry isfurther configured to display a display area of the second video and adisplay area of the first video in a same size on the display device. 5.The communication terminal of claim 2, wherein the second video is anequirectangular projection video obtained by imaging 360-degreesurroundings around the another communication terminal or the mobileapparatus.
 6. The communication terminal of claim 3, wherein thecircuitry is further configured to receive the first video includingone-side video viewed from one imaging direction and another-side videoviewed from another imaging direction, different from the one imagingdirection, the one-side video and the another-side video beingtransmitted by the another communication terminal or the communicationdevice, and wherein the circuitry is further configured to display theone-side video, the another-side video, and the second video on thedisplay device.
 7. The communication terminal of claim 3, wherein thecircuitry is further configured to display the first video as beingsuperimposed on the second video.
 8. The communication terminal of claim7, wherein the circuitry is further configured to: generate a duplicatevideo obtained by generating a duplicate of a partial angle of view ofthe second video; and display the duplicate video as being superimposedon the second video.
 9. The communication terminal of claim 8, whereinthe circuitry is configured to: receive an instruction for changing aposition of the first video or the duplicate video in the second video;and display the first video or the duplicate video at a positionindicated by the received instruction for changing the position.
 10. Thecommunication terminal of claim 9, wherein the circuitry is furtherconfigured to receive an instruction for deleting the duplicate video,an instruction for minimizing the duplicate video, or an instruction forgenerating a new duplicate video, wherein the circuitry hides theduplicate video when the circuitry receives the instruction for deletingthe duplicate video, wherein the circuitry minimizes a display size ofthe duplicate video when the circuitry receives the instruction forminimizing the duplicate video, and wherein the circuitry displays thenew duplicate video when the circuitry receives the instruction forgenerating the new duplicate video.
 11. The communication terminal ofclaim 1, wherein the circuitry is further configured to only one of thefirst video and the second video on the display device according to auser operation.
 12. The communication terminal of claim 5, wherein whenthe communication terminal starts communication with the anothercommunication terminal or the communication device, the circuitry isfurther configured to rotate the second video and display the firstvideo when the second video is rotated by a specific degree.
 13. Anon-transitory computer-readable medium storing a program for causing acommunication terminal, communicable with another communication terminalmounted on a mobile apparatus or with a communication device of themobile apparatus via a network, for transmitting operation instructioninformation for controlling the mobile apparatus to the anothercommunication terminal or the communication device to perform a methodcomprising: receiving a first video transmitted by one of the anothercommunication terminal and the communication device and a second videohaving a wide-angle of view captured by a wide-angle image capturingapparatus, the wide-angle image capturing apparatus being connected toone of the mobile apparatus and the another communication terminal orbeing configured as a single unit with one of the mobile apparatus andthe another communication terminal; displaying, on a display device, atleast one of the first video and the second video that are received; anddisplaying, on the display device, a controller for controlling anoperation of the mobile apparatus as being superimposed on the at leastone of the first video and the second video.
 14. A display methodperformed by a communication terminal, communicable with anothercommunication terminal mounted on a mobile apparatus or with acommunication device of the mobile apparatus via a network, fortransmitting operation instruction information for controlling themobile apparatus to the another communication terminal or thecommunication device, the method comprising: receiving a first videotransmitted by one of the another communication terminal and thecommunication device and a second video having a wide-angle of viewcaptured by a wide-angle image capturing apparatus, the wide-angle imagecapturing apparatus being connected to one of the mobile apparatus andthe another communication terminal or being configured as a single unitwith one of the mobile apparatus and the another communication terminal;displaying, on a display device, at least one of the first video and thesecond video that are received; and displaying, on the display device, acontroller for controlling an operation of the mobile apparatus as beingsuperimposed on the at least one of the first video and the secondvideo.